Disk space shortage wreaks havoc on a domain controller

One day our Windows 2003 domain controller ran out of disk space, and a slew of problems ensued. Apparently, Active Directory synchronization failed due to insifficient space, and the event log was full of messages like this:

Event Type: Warning
Event Source: NETLOGON
Event Category: None
Event ID: 5705
Description:
The change log cache maintained by the Netlogon service for  database changes is inconsistent. The Netlogon service is resetting the change log.

Event Type: Warning
Event Source: W32Time
Event ID: 26
Description:
Time Provider NtpClient: The response received from domain controller  has a bad signature. The response may have been tampered with and will be ignored.

Event Type: Error
Event Source: NETLOGON
Event ID: 5805
Description:
The session setup from the computer  failed to authenticate. The following error occurred: 
Access is denied. 

Event Type: Warning
Event Source: LSASRV
Event Category: SPNEGO (Negotiator) 
Event ID: 40960
Description:
The Security System detected an authentication error for the server cifs/.  The failure code from authentication protocol Kerberos was "The attempted logon is invalid. This is either due to a bad username or authentication information. (0xc000006d)".

Event Type: Error
Event Source: Kerberos
Event ID: 4
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/.  The target name used was ldap//@. This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named  machine accounts in the target realm (), and the client realm.   Please contact your system administrator.

Event Type: Error
Event Source: NETLOGON
Event ID: 3210
Description:
This computer could not authenticate with \\, a Windows domain controller for domain , and therefore this computer might deny logon requests. This inability to authenticate might be caused by another computer on the same network using the same name or the password for this computer account is not recognized. If this message appears again, contact your system administrator.

Even after freeing some space, problems continued, since Active Directory database was damaged. The solution was to reset computer domain password using netdom, as documented in Microsoft KB article 260575

What does "localhost" mean in named.conf?

Configuring BIND 9, run across an interesting issue. What does the following excerpt from named.conf mean:

options {
        listen-on port 53 { localhost; };
        ...
};

I thought this tells BIND to listen only on loopback adress (127.0.0.1). After all, this what "localhost" usually resolves to. To my great surprise, I've found that BIND is listening on all network interfaces. As it turns out, in the context of BIND configuration, localhost "Matches the IPv4 and IPv6 addresses of all network interfaces on the system." Go figure! The correct configuration is as follows:

options {
        listen-on port 53 { 127.0.0.1; };
        ...
};

Masquerading multiple IPSec connections on a Linux router

Our company network is set up in a rather standard way. We have a Linux router connected to an ISP with one network card, and to the local network with another one. The local network uses the reserved IP range 192.168.1.0-192.168.1.255. The router does Network Address Translation (NAT), also known as IP Masquerade, to allow the internal hosts transparent access to the Internet. This type of configuration is widely used and is well documented, in Linux 2.4 NAT HOWTO or Linux IP Masquerade HOWTO to name a few sources.

IP masquerade works well for TCP and UDP, but how about some more exotic IP protocols? Not that easy when it comes to IPsec VPNs. We need to connect to a customer's site that uses Contivity VPN from Nortel. This is an IPsec-based product. Unfortunately, masquerading several IPsec connection through one router is a non-trivial task that Linux is currently not capable of. IPsec connection involves a handshake over UDP, after which the data is transmitted over IP protocol 50 (Encapsulating Security Payload, ESP). Since there is no connection tracking support for IP protocol 50 in the Linux kernel, only one internal client can connect to a remote IPsec VPN server at any time, because the kernel cannot tell one connection from another. This was not acceptable to us, since we need several people to work with the customer's VPN simultaneously.

NAT Traversal feature of IPsec protocol is supposed to resolve this problem, however, for some reason, NAT Traversal didn't work with the customer's VPN server. Attempts to persuade the customer's IT personnel to look into the issue and enable NAT Traversal proved unsuccessful (they either didn't understand what the problem was, or didn't care, or both). We had to resolve the issue ourselves.

After some thought, the solution was found. First it involved getting several external IP addresses from our ISP, as many as many people we needed to work with the customer's VPN. Luckily, it wasn't many, just five. The idea then was to route the IPsec traffic from the five internal clients through separate external IP addresses. This will allow the kernel to keep track of each connection separately and not mix them up. Here's how I achieved this using iptables.

For the sake of example, I'll assume that our external IP numbers were 1.1.1.1, 1.1.1.2, 1.1.1.3, 1.1.1.4 and 1.1.1.5. All these numbers were bound to our external network interface as aliases. Assuming our external interface is called eth1, these new addresses were assigned to alias interfaces eth1:1, eth1:2, etc. up to eth1:5. Also for the sake of example, assume that the client computers that need to talk to the customer's VPN have internal IP addresses 192.168.1.1 - 192.168.1.5 and that the customer VPN server's address is 2.2.2.2.

With this setup, I added the following iptable rules to NAT table:

iptables -t nat -A POSTROUTING -s 192.168.1.1 -d 2.2.2.2 -j SNAT --to-source 1.1.1.1
iptables -t nat -A POSTROUTING -s 192.168.1.2 -d 2.2.2.2 -j SNAT --to-source 1.1.1.2
...
iptables -t nat -A POSTROUTING -s 192.168.1.5 -d 2.2.2.5 -j SNAT --to-source 1.1.1.5

This tells the kernel the following: right after routing (POSTROUTING chain), if the packet is coming from 192.168.1.x (a VPN client) to 2.2.2.2 (the VPN server), masquerade it using source address 1.1.1.x.

An important note is that these rules should precede the rules that masquerade the entire network. I have a rule like this:

iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth1 -j MASQEURADE

and it is added to the POSTROUTING change after the special rules above.

The above technique allows to masquerade multiple IPsec (or indeed any IP protocol) connections when NAT Traversal is not available. This said, I would much prefer if NAT Traversal worked and saved me the headache.

Эпос древних укров

История украинского народа уходит корнями в глубокую древность. Она не начинается с Оранжевой Революции, как наивно полагают некоторые на Западе. Она не начинается с Переяславской Рады, как хотели бы думать в Москве. Она не начинается даже с Кия, Щека и Хорива, как полагают некоторые ограниченные историки в Украине. Установлено, кто украинцы ведут свое происхождение от древнейшего народа укров, проживавшего на территории современной Украины уже в 4-м тысячелетии до н.э. К сожалению, в результате ряда катастроф как природно-геологического, так и социально-политического характера, литературные и культурные памятники укров были полностью утрачены. Однако нет на свете ничего, что было бы потеряно навеки (кроме Халезмийского собора - см. Д. Адамс, "Жизнь, Вселенная и все остальное", М., Аст, 2002, стр. 287). Достижения современной статистической историографии позволили буквально по крупице воссоздать и расшифровать фрагмент эпоса древних укров, публикуемый ниже. Работы по расшифровке продолжаются, и мы надеемся, что вскоре эпос будет полностью восстановлен.

Дальше...

SharePoint Server: Audiences based on group membership problem

SharePoint Portal Server is a great tool. One of its powerful features is audiences. Audience is a group of SharePoint server users to which the publishers can target the server's content. That is, certain items are shown only to certain audience members. For example, the server's home page may contain links to Development and Support group sites. Members of Development group only see the link to the Development site, and members of Support group only see the link to the Support site.

One of the ways to define an audience membership is via Windows security group. One can say that an audience consists of all members of a security group. This is often very convenient, as security groups often already reflect the organization's structure. However, setting up audiences on our company's SharePoint server I found that this simply didn't work. I set up an audience to consist of members of a security group that had 7 members. I found that the audience had only 1 member. I tried it with other groups - the results were utterly inconsistent. With some groups it worked, with some a few members were missing, with some all members were missing, and no clear pattern was seen.

After a long investigation the pattern was revealed. A security group member wouldn't make it to the group-based audience if the group was his/her primary group. It is a bug in SharePoint server, described in this blog post by Craig Gemmill for SharePoint Server 2003. I am using SharePoint Server 2007 and the bug is still present.

Apparently, it has to do with the fact that the user's primary group is not listed in its memberOf Active Directory attribute. The primary group's members attribute does not include the members with Primary Group attribute either. There are valid reasons for this design, described in this Microsoft KB article, but SharePoint Server, apparently, doesn't take Primary Group into account.

Thus the only solution was to reset the Primary Group of all users to Domain Users.

Using deadlines to install Windows updates on unattended servers

WSUS is a great tool to distribute and install Microsoft Updates. As long as the administrator approves the updates, an "Updates are ready" notification will pop up on each user's screen, and the users will be prompted to proceed with installing the updates.

But what about unattended servers? No-one ever logs into their consoles and thus there's nobody to install the updates. Having an administrator log in to each server to install updates is just too tedious. Luckily, this isn't necessary. Using deadlines, one can have the updates install on unattended servers automatically. Here's how I do it:

1. I've created a computer group in WSUS called "Servers" and added all servers to it.
2. Whenever I approve an update, I make a separate approval for the "Server" group with a deadline.
3. When the deadline expires, updates are installed automatically.

A thing to note here is that most of the time a server will reboot after the updates are installed. You don't want your servers to reboot in the middle of the day. Therefore, set the deadline to off-hours.

Always set the deadline more than 22 hours in the future. This is because Automatic Updates service is checking for new updates by default every 22 hours. Imagine what happens if during the day, say at 12am you set the deadline for 11pm today. By 11pm a server may not have checked for new updates yet, and it will not know a new update with a deadline is available. Next day, say at 10am it will check, find the update and see that the deadline is already past due. It will then immediately install the update and reboot right when your users are busy using it. To avoid such unfortunate scenario, allow sufficient time for Automatic Updates to find the update before the deadline expires.

Устанавливаем Lingvo 12

Наша компания приобрела электронный словарь Лингво 12. ABBYY Software предлагает для своего продукта удобную схему лицензирования - concurrent licenses. Покупая какое-то количество concurrent licenses, пользователь получает право запускать столько копий Лингво в сети одновременно, причем не имеет значения на каких компьютерах. Для нас это было очень удобно, т.е. из более ста сотрудников нашей компании, одновременно Лингво не понадобится больше, чем двадцати.

Технически работает эта схема лицензирования так. На сервере устанавливается Lingvo License Manager. В него вводятся и активируются все лицензии. Клиентские машины получают лицензию от сервера при запуске Лингво.

На бумаге все хорошо. На практике обнаружилось, что при запуске Лингво на одном компьютере License Manager показывает, что все concurrent лицензии заняты этим компьютером, и ни один другой компьютер запустить Лингво уже не может. Весьма неприятный сюрприз. В интернете обнаружили описание такой же проблемы (http://forum.lingvo.ru/actualthread.aspx?tid=72935) и совет обратиться в службу поддержки. Написали на support@abbyy.ua. Служба поддержки ответила, поинтересовалась номером нашей лицензии, и действительно прислала нам обновление на License Manager, исправляющее эту ошибку. Таким образом при доработке напильником система все-таки работает.

Несколько неприятной, однако, оказалась следующая новость в Лингво 12. В предыдущих версиях Lingvo был LvAgent, который сидел в System Tray и позволял вызвать Lingvo по Ctrl-Ins-Ins. При закрытии самого Lingvo LvAgent продолжал работать. Теперь все немного не так. По-прежнему LvAgent может запустить Lingvo, но теперь при закрытии Lingvo стандартными способами (кнопочка "x" или Alt-F4), сам Lingvo продолжает работать и, что особенно важно при concurrent лицензировании, занимать лицензию. Т.о. стоит запустить Lingvo, как он занимает лицензию даже после закрытия. Есть, правда, в меню пункт "Выход", который действительно закрывает Lingvo - но вместе с LvAgent.

В свете этого моя рекомендация такая - в настройках на вкладке "Общие" снять флажок "Отображать значок ABBYY Lingvo в области уведомлений на панели задач". В этом случае LvAgent не используется и закрытие Lingvo действительно его закрывает. Но вызов по Ctrl-Ins-Ins будет невозможен.

И наконец, странная история, приключившаяся с инсталляцией Лингво. После установки License Manager, по рекомендации производителя, инсталляция самого Лингво была помещена на сетевой диск. Вскоре несколько пользователей пожаловались, что они не могут установить Лингво себе на компьютер, используя эту сетевую инсталляцию: инсталлятор сообщает "Внутренняя ошибка 2349" и прекращает работать. Выяснилось, что по какой-то причине часть файлов из инсталляции получили дату изменения 01.01.1857 (как это произошло - загадка). При копировании этих файлов инсталлятор пытался дать установленным копиями такую же дату, что заканчивалось ошибкой. После исправления даты файлов проблема разрешилась.