Showing posts with label sysadmin. Show all posts
Showing posts with label sysadmin. Show all posts

Saturday, November 27, 2010

Black logon screen on Windows Server

One day I was logging on to one of our Windows servers and was greeted by the following:

The screen was completely black except for the Windows Server logo. It's not a screen saver! Frankly, I freaked out. What kind of black screen of death that was? Then I realized the server wasn't dead. The screen was black, but fully functional. If you can see the little white vertical bar under the logo, that's the insertion point in the password entry field. So, with some blind typing I was able to log in. But what's the problem?

What do you know, it's a known one and has a Microsoft KB article 906510 dedicated to it. Turns out, it happens if for some reason the key HKEY_USERS\.Default\Control Panel\Colors gets reset to all zeros. The key allows to customize the colors used by the logon screen; in this case the colors are "customized" to all black! Why would it happen, I don't know. I suspect, what triggered the problem was the server running out of space on its system drive some time before. Windows servers have a tendency to tolerate this very badly, as I once wrote. The solution is to export this key from some other Windows server and import on the affected server.

Saturday, November 20, 2010

Windows tools: handle and taskkill

It's been a while since I posted on system administration! Here are two findings - probably trivial, but new for me. 

Who's locking?

Say you can't open, move or delete a file on a Window system because it is locked by another process. How do you find out which process is responsible? To handle this question use Handle, a utility from Sysinternals suite. Basic usage:

handle <file name>

<process name> can be the full file name or any part of it. There are a few handy options, e.g. -c to close the handle in question. Handle is pretty much an equivalent of fuser(1) on Linux.

Killing hung service

It happens from time to time that a service gets stuck in "Stopping" or "Starting" state. How can you kill it without rebooting the entire machine? On Windows Server 2003 or 2008 the built-in taskkill command-line utility can help. Use the following syntax:

taskkill /fi "SERVICE eq <servicename>" /f

where <servicename> is the name of the service that has hung (you can see it under "Service name" in service's properties window in Services console). For example:

taskkill /fi "SERVICE eq TlntSvr" /f

kills Telnet service, if it happens to get stuck. Taskkill has many options, e.g. it can kill a process by PID with /pid switch. The /f switch forces termination of an unresponsive process.

 

 

 

Saturday, July 10, 2010

Redirecting a Samba share to another server

In this post I'll describe how to configure Samba to redirect a share to another server. A typical example when this might be useful is as follows. You have a share, say \\server\work that you need to migrate to new hardware, so that it becomes \\newserver\work. However, you don't want all your users to remember that the files are at a new location. You'd like your users to continue using the old address \\server\work and be transparently redirected to \\newserver\work. This is possible with MS DFS (distributed filesystem) support via msdfs proxy directive. It seems like a more obscure directive not very well documented and described, so I'll try to report my experience with it.

The first thing to do is to enable MS DFS in Samba on the global level by specifying:


host msdfs = true

in the [global] section of smb.conf.

Then in the section for your share, you remove the path directive and insert two MS DFS-related directives, msdfs root and msdfs proxy:


[work]
msdfs root = true
msdfs proxy = \newserver\work

Note a few things about this configuration:
  1. You MUST NOT have the path directive in the share definition
  2. The value of msdfs proxy directive is the new location of the share to redirect to; notice though, that's a single backslash in the beginning of the value, NOT the double backslash you could have expected!

That's all, now \\server\work transparently redirects to \\newserver\work. Clients may need to be rebooted to start working properly with the redirected share. In my experience, clients can't access the share right after the reconfiguration, but it starts working after some time (not sure how much time) or after the client reboots.

Per-client-IP configuration for VSFTPD

Recently I needed to configure VSFTPD FTP server to behave differently depending on the client IP address. Specifically, the task was to limit the number of connections to no more than 3 per client IP, but lift this restriction for one specific IP address (which was our own office address). The per-IP limit itself is easy to implement with max_per_ip configuration setting (see VSFTPD FAQ). The trick was specifically in making our office address the exception to this limit.

While per-user configuration is easy to implement with user_config_dir setting (and it is described in the FAQ), per-IP configuration is less obvious, so I thought I would document my findings.

The path to per-IP configuration lies via interaction with tcp_wrappers. Just in case you don't know, tcp_wrappers is a tool for filtering incoming network requests, and VSFTPD makes use of it. So, the first thing to do, is to enable tcp_wrappers support in VSFTPD, by making this setting in vsftpd.conf:


tcp_wrappers=YES

Then proceed to enter the default configuration in the main vsftpd.conf file. In my case, I set max_per_ip to limit connections per IP address:


max_per_ip=3

The next step is to configure tcp_wrappers. Add a line similar to the following to /etc/hosts.allow file:


vsftpd:1.2.3.4/255.255.255.255:setenv VSFTPD_LOAD_CONF /etc/vsftpd/vsftpd.conf.special

This must be a single line, even though Blogger may wrap it. It consists of 3 fields, separated bu the ':' character. The first must be vsftpd. The second contains your "special" IP address for which you want to enable a special configuration. As shown above, it may include a netmask so that you can apply the special configuration to a subnet (see man page for hosts.allow for details). Finally, the third field instructs tcp_wrappers to set an environment variable VSFTPD_LOAD_CONF to the name of a configuration file that will apply to your special IP address(es). The name of the variable must be exactly this; the value may be any filename you like. Once tcp_wrappers set this variable, VSFTPD will load the configuration file specified in it for the IP address(es) from the second field.

So, in the additional configuration file (in my example, /etc/vsftpd/vsftpd.conf.special) you can override everything you want for your special addresses. In my case I just needed to turn off per-IP restiction:


max_per_ip=0

That's it. I find it rather convoluted, but it works and achieves a useful purpose.

Tuesday, November 03, 2009

Administration under Windows 7

I received a new computer with Windows 7, and here are the first impression of doing system administration tasks from this new OS:

1. Windows Server 2003 admin pack no longer works, need to download Remote Server Administration Tools for Windows 7 from here.

2. Even then, Dial-in tab is not available. There is an old workaround from Windows XP time that still works though. Its descrption can be found in many places, including here, but here's brief recap:

  • Copy/Paste the following to a Dialin.reg file:
    Windows Registry Editor Version 5.00
    
    [HKEY_CLASSES_ROOT\RasDialin.UserAdminExt]
    @=""
    
    [HKEY_CLASSES_ROOT\RasDialin.UserAdminExt\CLSID]
    @="{B52C1E50-1DD2-11D1-BC43-00C04FC31FD3}"
    
    [HKEY_CLASSES_ROOT\RasDialin.UserAdminExt.1]
    @=""
    
    [HKEY_CLASSES_ROOT\RasDialin.UserAdminExt.1\CLSID]
    @="{B52C1E50-1DD2-11D1-BC43-00C04FC31FD3}"
    
    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MMC\NodeTypes\{19195a5b-6da0-11d0-afd3-00c04fd930c9}\Extensions\NameSpace]
    "{B52C1E50-1DD2-11D1-BC43-00C04FC31FD3}"="RAS Dialin - User Node Extension"
    
  • Merge the Dialin.reg file with your Windows XP registry, or run regedit /s dialin.reg.
  • Copy 3 DLLs from a Windows Server 2003 domain controller's system32 directory to your computer's system32 directory: mprsnap.dll, rasuser.dll, rtrfiltr.dll and register rasuser.dll with regsvr32. This can be automated by the following sequence of commands:
    CD /D %SystemRoot%\System32
    copy \\ServerName\Admin$\System32\mprsnap.dll *.*
    copy \\ServerName\Admin$\System32\rasuser.dll *.*
    copy \\ServerName\Admin$\System32\rtrfiltr.dll *.*
    regsvr32 rasuser.dll
    

3. There's no Exchange System Manager for Windows 7 (at least for Exchange 2003). The one from Exchange 2003 doesn't work. The ESM for Vista works, but its installer fails to install it on Windows 7 (it strictly checks for Vista). Some generous soul has hacked the install to work on Windows 7 and made it available for download from this page: http://nwcalc.web44.net/Win7/. See the discussion here

Tuesday, October 13, 2009

Samba shares inaccessible after power failure

A test Samba server (domain member with ADS security) suffered a crash due to power failure, and after that its shared became inacessible: when Windows clients tried to access them, password dialog was shown and the correct password was not accepted.

Samba log showed the following:

[2009/10/08 13:22:25, 1] smbd/sesssetup.c:reply_spnego_kerberos(316)
  Failed to verify incoming ticket with error NT_STATUS_LOGON_FAILURE!

Winbind log showed the following

[2009/10/08 13:13:45, 5] nsswitch/winbindd_user.c:winbindd_getpwnam(353)
  Could not parse domain user: lz

The problem was resolved by removing all *.tdb files in /var/lib/samba.

WARNING. This was a test server, so removing the files were not a problem. If it were a production server, removing windbindd_idmap.tdb would destroy the mapping of Windows and Unix users, and thus all file permissions on Samba shares would be screwed up. On a production server one would try to rescue at least windbindd_idmap.tdb.

Monday, April 13, 2009

Network access blocked to a server

A domain controller running Windows 2003 Server SP1 ran out of disk space. When I tried to connect to it to clean up the disk, I found that the server was not accessible from network. Even ping didn't work. I logged in to the server's console and found the following events in the system log:

Source: IPSec
Event ID: 4294
Description:
The IPSec driver has entered Secure mode. IPSec policies, if they have been
configured, are now being applied to this computer.

Event type: Error
Source: IPSec
Event ID: 4292
Description:
The IPSec driver has entered Block mode. IPSec will discard all inbound and outbound 
TCP/IP network traffic that is not permitted by boot-time IPSec Policy exemptions. 
User Action: To restore full unsecured TCP/IP connectivity, disable the IPSec 
services, and then restart the computer.  For detailed troubleshooting information, 
review the events in the Security event log.

The explanation was found in the following Microsoft KB article: http://support.microsoft.com/kb/912023. Apparently, the IPsec security policy registry key got corrupted, and IPsec panicked and blocked all access to the computer. The solution was, as described in the KB article in detail, to remove the following registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\IPSec\Policy\Local.

Saturday, April 11, 2009

Setting ip IPsec tunnel from Linux to Cisco PIX

In this post I am going to put down my experience setting up a IPsec tunnel from a Linux router to a Cisco PIX device. I'll explain the setup, the solution, and the pitfalls encountered along the way.

Setup

We have a LAN using IP subnet 192.168.0.0/22, connected to the Internet through a Linux router, whose external IP address is 80.1.1.1 (address made up for the sake of example, of course). The goal was to connect to a network within customer's LAN through IPsec tunnel. The destination net was 194.1.1.2/32 (yes, the netmask here is 32, since this actually was a single host; but I'd like to talk about it as a subnet, because in the case of a true subnet our solution would be the same, just netmask would have less bits). Even though the destination's IP address is "external-looking", it is blocked off the Internet by a firewall and is only reachable through IPsec via an external gateway whose address is 194.1.1.1. The following crude ASCII art diagram illustrates the setup:

                              80.1.1.1     
   Our LAN     |------------|/              
---------------+Linux router+--------------|
192.168.0.0/22 |------------|              |
                                           |
            IPsec tunnel                   |
    ---------------------------------------|
    |194.1.1.1
    |         \|---------|  Dest. subnet
    |----------+Cisco PIX+----------------
               |---------|  194.1.1.2/32

Additional complication was that our IP address range 192.168.0.0/22 was in use in the customer's LAN, thus we could not allow our packets to go into their network with their original source IP addresses. To circumvent this, our first step was to set up SNAT to masquerade all our outgoing packets as coming from IP address 172.16.1.1.

The Cisco PIX was managed by the customer, and thus outside of our control. I was responsible to set up our Linux router.

Solution

Now I will jump forward and describe the solution that worked.

Some theory on IPsec and indispensable guidance on setting up IPsec in Linux can be found at Linux IPsec HOWTO. The solution below was based on the information obtained from the HOWTO.

SNAT

As mentioned above, our first step was to set up SNAT for our packets going off to the customer's net:

iptables -t nat -A POSTROUTING -d 194.1.1.2/32 -j SNAT --to-source 172.16.1.1

Note that this is independent of IPsec. It just tells the kernel to masquerade any packets going out to 194.1.1.2/32 as coming from source address 172.16.1.1 (which does not need to correspond to any physical network interface of the router). Before any tunnel was set up, I could test the SNAT by pinging 194.1.1.2 and watching the source address of outgoing packets with tcpdump.

Kernel security policy

The next step is to set up kernel's security policy. Simply put, we need to tell the kernel that whenever it sees a packet going to 194.1.1.2/32 it has to send it through tunnel that runs from 80.1.1.1 (our external address) to 194.1.1.1 (customer's external address). This is achieved with the following script:

#!/bin/sh

SRCNET="172.16.1.1/32"
DSTNET="194.1.1.2/32"
TUNNEL_LOCAL="80.1.1.1"
TUNNEL_REMOTE="194.1.1.1"

/sbin/setkey -c  >/dev/null 2>&1 << EOF

spdadd $SRCNET $DSTNET any -P out ipsec
            esp/tunnel/$TUNNEL_LOCAL-$TUNNEL_REMOTE/require
            ;

spdadd $DSTNET $SRCNET any -P in ipsec
            esp/tunnel/$TUNNEL_REMOTE-$TUNNEL_LOCAL/require
           ;
EOF

Note that we only tell the kernel to use ESP and not AH. Many sources would tell you to use both, and this is a good thing to do; it's just that in our case, for some reason, the customer insisted on using ESP only. Also note that the SRCNET is set to 172.16.1.1/32, as the SNAT we set up earlier will modify the packets' source addresses before IPsec sees them. Had we not used SNAT, SRCNET would have been 192.168.0.0/22.

Now the kernel knows when to use the tunnel, but to create the tunnel a security association should be set up between our router and the remote Cisco box. This is the job of racoon daemon, and I'm moving on to describe its configuration.

Racoon and ISAKMP

Racoon speaks ISAKMP protocol in order to create the tunnel. A detailed description if ISAKMP is outside the scope of this post, but it is important to know that creation of the tunnel proceeds in two phases. At phase 1 racoon negotiates a preliminary secure connection to the remote site called ISAKMP Security Association (SA). At phase 2, using the existing ISAKMP SA, it is able to privately talk to the remote site and negotiate the parameters of the IPsec tunnel itself. At phase 2 the actual IPSec Security Association (SA) is created.

There are many details of ISAKMP that racoon needs to negotiate with the remote end of the tunnel. Description of them all is outside of our scope now. Our customer wanted to use the following parameters:

Phase 1:

  • Authentication Method: RSA-Key
  • Diffie-Hellman Group: 5 (1536 bit)
  • Encryption Algorithm: AES-256
  • Data Integrity Algorithm: SHA-1
  • Use aggressive mode: No
  • Lifetime: 86400s (1 day)

Phase 2:

  • Encapsulation (ESP or AH): ESP
  • Encryption Algorithm: AES-256
  • Authentication Algorithm: SHA-1
  • Perfect Forward Secrecy: No
  • Lifetime: 3600s (1 hour)

Let's to go over these requirements and try to make sense of them.

Phase 1:

  • Authentication Method: RSA-Key. There are several possible authentication methods, such as pre-shared key or RSA keys (or, rather, X509 certificates). We are going to use certificates.
  • Diffie-Hellman Group: 5 (1536 bit). Diffie-Hellman algorithm is used to exchange session keys between the IPsec peers. The "group" is one the algorithm's parameters. The important thing is that both peers use the same value.
  • Encryption Algorithm: AES-256. This just means that out of the multitude of encryption algorithms they want to use AES with 256-bit long keys.
  • Data Integrity Algorithm: SHA-1. Similarly this just means that of all the hash algorithms they want to use SHA-1.
  • Use aggressive mode: No. Aggressive mode is an optional feature of Phase 1 negotiation that "reduces the number of round-trips at the expense of not providing identity protection" (RFC2408). They don't want to use it. I guess they care about identity protection.
  • Lifetime: 86400s (1 day). This means that the ISAKMP SA that we establish at Phase 1 will have lifetime of 1 day, and after that will have to be renegotiated.

Phase 2:

  • Encapsulation (ESP or AH): ESP. IPsec uses two protocols, ESP and AH. For some reason, they only want to use ESP (this is why when setting up kernel security policy above I told it to use ESP only).
  • Encryption Algorithm: AES-256. Just using the same encryption algorithm as in Phase 1.
  • Authentication Algorithm: SHA-1. Just using the same hash algorithm as in Phase 1.
  • Perfect Forward Secrecy: No. Perfect Forward Secrecy is an optional feature of Phase 2 negotiation, which means that "disclosure of longterm secret keying material [RSA private key in our case] does not compromise the secrecy of the exchanged keys from previous communications". For whatever reason, they don't want to use it.
  • Lifetime: 3600s (1 hour). This means that the IPsec SA that we establish at Phase 2 will have lifetime of 1 hour, and after that will have to be renegotiated, so that if anyone is eavesdropping on us, they won't have much time to brute-force our keys.

The above requirements translate into the following racoon config that can be added to /etc/racoon/racoon.conf. The comments in the config below explain how config directives correspond to the requirements.

# The following section tells racoon how to conduct Phase 1 
# negotiation to the remote peer 194.1.1.1
remote 194.1.1.1
{
  # We are NOT using aggressive mode, therefore the mode is "main"
  exchange_mode main;
  # The peer identifies themselves by their IP address. See below in "Pitfalls" 
  # section on why this directive was necessary.
  peers_identifier address;
  # We will not verify the peer's certificate. See below in "Pitfalls" 
  # section on why this directive was necessary.
  verify_cert off;
  # The following line specifies the location of our certificate and private key
  certificate_type x509 "ipsec.cer" "ipsec.key";

  proposal {
    # for encryption algorithm we will use AES 256
    encryption_algorithm aes 256;
    # for hash algorithm we will use SHA1
    hash_algorithm sha1;
    # we will authenticate with X509 certificate (called rsasig here)
    authentication_method rsasig;
    # We will use Diffie-Hellman group 5
    dh_group 5;
    # Lifetime of phase 1 association will be 24 hours
    lifetime time 86400 sec;
  }
}

# The following section tells racoon how to conduct Phase 2
# negotiation for packets exchanged between subnets 172.16.1.1/32 and
# 194.1.1.2/32. Notice that here we use "internal" source and destination
# subnet addresses, and NOT tunnel endpoint addresses.
sainfo address 172.16.1.1/32 any address 194.1.1.2/32 any
{
  # Lifetime of Phase 2 association will be 1 hour
  lifetime time 1 hour ;
  # for encryption algorithm we will use AES 256
  encryption_algorithm aes 256 ;
  # for authentication algorithm we will use SHA1
  authentication_algorithm hmac_sha1;
  # For compression algorithm we will use "deflate". This wasn't in the specs,
  # but it's the default
  compression_algorithm deflate ;
}

Pitfalls

IPsec is a complex protocol and many are the pitfalls on the road to a successful IPsec tunnel.

Parameters

Parameters offered by the initiator side (such as encryption algorithms, lifetimes, Diffie-Hellman group, etc.) must be exactly the same as expected by the remote side. In theory ISAKMP is supposed to negotiate the parameters, but in practice I found that even the slightest difference led to failed negotiation. The problem is aggravated by the fact that different implementation of IPsec (in our case that of Linux and Cisco PIX) use different configuration syntax and slightly different terminology, so that translating the requirements of Cisco PIX into the configuration language of racoon is not always straightforward.

Identification

In IPsec each party identifies itself to the other party and presents credentials to prove their identity. The protocol allows for several forms of identification, such as:

  • By IP address (i.e. a peer says "I am the owner of this IP address")
  • By DNS name (i.e. a peer says "I am the owner of this DNS name")
  • By distinguished name from an X509 certificate (i.e. a peer says "My DN is as follows")
etc. When you configure a tunnel, you need to specify which form of identification you use. When your credential is an X509 certificate, the best choice is identification by distinguished name, because the certificate is exactly the kind of credential that proves that the DN belongs to you. This form of identification is achieved by the following line in racoon configuration:

my_identifier asn1dn; 

However, our customer configured its Cisco to identify itself by IP address. How does one prove with an X509 certificate that the IP address is his? Well, racoon expects that the certificate in this case will have an subjectAltName attribute with the value "IP:xx.xx.xx.xx" (where xx.xx.xx.xx is the peer's IP address). Unfortunately, our customer's certificate did not have such attribute. To work around, I had to turn off the certificate validation with "verify_cert off;" directive.

Tunnel stability

Unfortunately, the tunnel proved to be unstable. Approximately once in several days the remote end would terminate the ISAKMP association. This is not a problem in itself, since racoon is capable of re-negotiating the association when needed. However, apparently the remote end also terminates the IPsec association, but racoon does not notice this; thus it continues to use the IPsec association that the remote end no longer accepts and no data comes through.

In about an hour's time racoon will expire the IPsec connection and re-negotiation a new one, so the tunnel will be re-established. If you can't wait, you can force the renegotiation. One way to do it is to delete all security policy entries with setkey -F command. Another is to remove the IPsec association with racoonctl command (see manpage for details).

Saturday, December 27, 2008

Как Касперский тормозит почту

С некоторых пор стал наблюдать следующую картину на почтовом сервере: в очереди sendmail'а стоят письма, адресованные локальным пользователям, но не доставляются. Более того, письмо часто адресовано целой группе пользователей, но доставка затыкается на ком-то одном, и все, кто в списке после него, письмо не получают.

Посмотрим, нет ли блокировок почтовых ящиков:


# cd /var/spool/mail
# ls *.lock
lz.lock
# 

Ага, один ящик заблокирован (здесь я описываю на примере моего ящика lz, хотя в жизни это были ящики других пользователей). Посмотрим, кто его использует:

# fuser -v lz
                     USER        PID ACCESS COMMAND
lz:                  lz        20384 F.... pop3
                     lz        20476 F.... procmail
# 

Похоже, процесс pop3-сервера занял ящик, поэтому procmail не может открыть его на запись. Проверим:

# lsof -p 20384 
COMMAND   PID USER   FD   TYPE   DEVICE     SIZE     NODE NAME
...
pop3    20384   lz    9uR  REG    253,3 11922701 16680817 /var/spool/mail/lz

(здесь число послее опции -p команды lsof - это PID, который вывела команда fuser; и я опустил все строки в выводе lsof, кроме интересующей меня).

Действительно, файл заблокирован на чтение (о чем говорит большая буква R в колонке FD). Но с какой стати процесс pop3 это сделал? Казалось бы, он должен выдать клиенту все сообщения и "отвалить". С помощью strace'а я посмотрел, чем был занят процесс pop3 - оказалось, периодически получает от клиента команды NOOP, и все. Зачем клиент посылает эти команды и не отключается? Я проверил поведение Outlook Express'а на своем компьютере - как и ожидалось, он подключался, забирал почту и отключался, т.е. не висел и команд NOOP не посылал.

Далее я заметил, что пользователи, чьи ящики таким странным образом блокировались, были все время одни и те же. Это побудило меня тщательно обследовать их компьютеры. И тут на сцене появился герой этой заметки - антивирус Касперского. На компьютерах всех пользователей, страдавших от блокировки, работал почтовый антивирус. Отключили почтовый антивирус - проблема ушла.

И черт с этим почтовым антивирусом, все равно почту проверяет ClamAV

Sunday, November 23, 2008

Forcing Sendmail to talk plain SMTP

While routinely checking out our mail queue I noticed a message stuck in it with an explanation I never saw before:

Deferred: 402 4.5.2 Error: command not recognized

Command not recognized? What is that supposed to mean? I ran sendmail -q -v manually and observed the following dialogue:

>>> 220 *******************************
<<< EHLO csltd.com.ua
>>> 402 4.5.2 Error: command not recognized
<<< QUIT
>>> 221 2.0.0 Bye

It looked like the remote mail server did not recognize Extented SMTP EHLO command. However, instead of replying with 500 error code (as it should when it does not recognize a command), it replied with 402 "temporary failure" code. Had it replied with the correct 500 code, our Sendmail would fall back to normal SMTP and retry with HELO command. But with 402 reply our Sendmail deferred the message.

Further analysis showed that the remote site deployed an overly clever Cisco firewall that filtered SMTP traffic. The firewall was configured (intentionally or not) to allow only plain SMTP commands. Anything else, including ESMTP EHLO, it replaced with XXXX. On top of that, the mail server software replied with an incorrect error code to the unknown command.

What a mess. How could I work around this? I had to force our Sendmail to talk plain SMTP to that particular remote server. This can be easily done with mailertable. Assume for the sake of example that the remote domain was brokensmtp.com. In /etc/mail/mailertable, I added the following entry:


brokensmtp.com       smtp:brokensmtp.com

The key here is that I specify smtp mailer instead of the default esmtp. Ran make in /etc/mail directory to rebuild the mailertable, and the issue was solved.

Friday, November 07, 2008

Windows 2003 Server cannot access Samba share

We migrated a certain application from a Windows XP workstation to Windows Server 2003 and faced the following issue. The Windows Server 2003 computer could not access a share on a remote Samba server that is running in security = server mode. Attempts to access the remote share resulted in this error message:

\\server is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out if you have access permissions.

The request is not supported.

For various reasons we could not change the Samba server security mode to ADS. A workaround, however, was found here.

On Windows Server, start Local Security Policies, navigate to Local Policies --> Security Options. Open the "Network Security: LAN Manager Authentication Level" setting and set it to "Send LM and NTLM responses".

After that, remote Samba share is accessible. No reboot necessary.

Saturday, October 25, 2008

"Failed to set servicePrincipalNames" Samba error

I was trying to join a Samba server to an Active Directory domain with a standard command:

net -U administrator ads join

Unexpectedly, the command failed with the following message:

Failed to set servicePrincipalNames. 
Please ensure that the DNS domain of this server matches the AD domain, 
Or rejoin with using Domain Admin credentials.
Disabled account for 'SERVERNAME' in realm 'DOMAIN' 

I was puzzled, since DNS domain name was correct, and I was using Domain Admin credentials.

After some searching, I came upon an advice here to check /etc/hosts. And indeed, the file contained an entry created by my distribution (Fedora) setup process which looked like:


SERVER_IP_ADDRESS SERVERNAME

Where SERVERNAME was a short name (without domain). Changing it to

SERVER_IP_ADDRESS SERVERNAME.DOMAIN

or just removing it resolved the issue. Rather weird.

Saturday, September 06, 2008

Forcing Domain Admins to be local admins on domain computers

All network administrators want to be able to administer user workstations remotedly. Often this requires the administrator to be a member of the local Administrators group on the user workstation. A simple way to ensure this is to have Domain Admins group to be a member of local Administrators group on each computer. Not surprisingly, this is the default situation on all Windows computer that are domain members.

However, if your users are local admins on their computers, sometimes you can find yourself in a situation when a clever or misguided user removed Domain Users from their local Administrators group. Also, sometimes you want some less privileged group than Domain Admins to be local admins on all computers - for example a group of network administrators that are assigned to manage user computers but are not eligible to administer the domain. In both cases you face a problem: how to ensure that a certain domain group is always a member of local Administrators group on each domain member computer?

Group policy to the rescue. Below I describe how to achieve this goal, as well as the pitfalls and shortcomings of this seemingly simple solution.

The key to the solution is to configure a feature in Group Policy called "Restricted Groups". In the following steps I will assume that you want to make Domain Admins part of local Administrators, but you can use any other group that you want.

Fire up Group Policy Management Console and locate your default domain policy (or any other Group Policy Object you want to use for this configuration). Right-click it and select edit. In the Group Policy Editor navigate to Computer Configuration\Windows Settings\Security Settings\Restricted Groups. Right-click and select "Add Group". Browse for your Domain Admins group. You should now arrive at the dialog box that looks like this:

First, in the top part of the dialog window, press "Add..." and add all the members of your Domain Admins group here. This seemingly extra step is necessary, because when the Group Policy is applied, it will change the membership of the group to the one specified here. If you don't list all your Domain Admins members here, you will wipe out your Domain Admins group! So, be careful and specify all relevant members here.

Now you are ready to do what you came here for. In the bottom part of the dialog, press "Add..." and type in Administrators group name. Note: do not use "Browse..." button, just type in the name. The dialog will now look like this:

Click OK and you are done. Once Group Policy applies to domain member computers, Domain Admins will become part of local Administrators group if not already there.

Caveats

Unfortunately, there are issues with this solution that you must be aware of. They all stem from the fact that Group Policy matches the Restricted Groups by name rather then SID; i.e. when the policy applies your Domain Admins will become a member of a group named Administrators; but it's not this name that makes this group special, it's its well-known SID. Let's see what implications this has.

For one, a malicious user may simply rename the local Administrators group (e.g. call it Admins), and it will completely defeat our trick, because Group Policy won't find a group named Administrators. This said, a truly malicious user will always find a way to prevent Group Policy from applying to his computer or to undo its effect after it applied. Thus we shouldn't even expect Group Policy to help overcome malicious users.

Next, consider international users. The local admin group is called Administrators in English version of Windows, but in other language versions the name is localized. In Russian version of Windows the group is called Администраторы, in Cyrillic. If you have both Russian and English systems in your domain, you can simply add both names to the "This group is a member of" list:

This will work, but has its own problems. Whenever Group Policy applies, it will find either Russian or English version of the Administrators name, depending on the system language. The other name will not be found, and a warning like the following will be logged to the Application system log.


Event Type: Warning
Event Source: SceCli
Event Category: None
Event ID: 1202
Date:  9/4/2008
Time:  1:41:37 AM
User:  N/A
Computer: 
Description:
Security policies were propagated with warning. 0x534 : No mapping between account names and security IDs was done.

Advanced help for this problem is available on http://support.microsoft.com. Query for "troubleshooting 1202 events".

Error 0x534 occurs when a user account in one or more Group Policy objects (GPOs) could not be resolved to a SID.  This error is possibly caused by a mistyped or deleted user account referenced in either the User Rights or Restricted Groups branch of a GPO.
...
[very detailed explanation follows]

That might be not a big deal on a workstation, but on domain controllers the Group Policy applies every 5 minutes, and the Application log will be flooded with these warnings, making it harder to spot more serious warnings, if they occur.

What can we do? Well, this policy doesn't need to apply to domain controllers, they don't even have local groups! So, you can do the following:

  1. create a new GPO;
  2. make the Restricted Groups configuration there and not in Default Group Policy;
  3. make the GPO apply to Domain Computers group only (which does not include domain controllers). For this add Domain Computers group in the "Security Filtering" section of Group Policy Management Console for this GPO and remove any other groups that were there, like this:

That gets rid of the flood of warning on domain controllers. An occasional warning in member servers' and workstations' logs can be tolerated, but if not, you have a way to get rid of them. For that you will need to create two separate GPOs and make them apply to either English or Russian systems with WMI Filters.

Using WMI Filters

So, back in Group Policy Management Console, create a new GPO. Let's call it "Russian language-specific policy". Make it apply to Domain Computers as described above. Now, under WMI Filters, in the Group Policy Management Console's left-hand tree, select "New" from the context menu to create a new filter with the following condition:

select * from Win32_OperatingSystem where OSLanguage = 1049

Your "New WMI Filter" dialog will look as follows:

Make the filter apply to the GPO by selecting the filter from the drop down list in the "WMI Filtering" section of Group Policy Management Console:

Voila, you have a GPO that applies only to domain computer with Russian version of Windows. Now you can configure Restricted Groups in it using the Russian name "Администраторы".

Repeat the same steps to configure "English language-specific policies" with a similar WMI Filter:

select * from Win32_OperatingSystem where OSLanguage = 1033

Now you can configure Restricted Groups in it using the English name "Administrators".

Somewhat complicated, but now you have different policies that apply to different language versions of Windows. Was it worth it? In this case probably not, but it's an interesting exercise and the technique may help you in other situations.

Monday, May 19, 2008

Corrupt Winbind IDMAP database

After a Samba server crashed due to a power failure, several users could no longer access it shares. When access was attempted, password dialog was shown and the correct password was not accepted. Strangely, the problem affected only some users, while others continued to work in Samba shares successfully.

The Samba server's /var/log/messages showed the following errors:

May 19 12:48:05 server winbindd[1929]:   tdb(/var/lib/samba/winbindd_idmap.tdb): 
tdb_rec_read bad magic 0x42424242 at offset=102664
May 19 12:48:05 server winbindd[1929]:   Error storing SID -> ID: Corrupt database

So, the problem was that windbindd_idmap.tdb file was corrupted. In this situation, the only sensible solution is to restore the file from a backup. If you don't have a backup, you are out of luck. Best you can do is remove the corrupted file, but it stores the mapping between Windows and Unix user IDs, and as the result file ownership and permission information on the Samba shares will be screwed up.

Tuesday, April 22, 2008

Windows computer can't get IP address from DHCP

Suppose you configured a Windows computer to obtain IP address automatically via DHCP, but it doesn't work. Ipconfig shows that IP address is 0.0.0.0 and running ipconfig /renew returs errors like "Unknown interface" or "RPC server not available".

The most likely reason for this: DHCP Client service is not running. It is started automatically by default, but some users stop it, because it is not needed when you have a static IP address. Just start the service and ipconfig /renew should get you your IP address.

Friday, April 18, 2008

Apache: How to enable CGI in one user's directory

Recently I had to configure Apache HTTPD server on a Fedora 7 box to allow CGI script execution in one particular user's public_html directory, yet not allowing it for all users.

Easy, I thought. First, Apache needs to know that a certain file is a CGI script and not a regular web page. Normally CGI scripts live in cgi-bin directory and Apache knows that anything in it is a CGI script by virtue of ScriptAlias directive. But how does it know a file outside of cgi-bin directory is a CGI script? Usually, by the file's extension. To tell Apache that any file with .cgi extension is a CGI script, uncomment the following line in the default httpd.conf (or add it if it's not there at all):


AddHandler cgi-script .cgi

Now Apache knows which files are CGI scripts. So far so good, but how do we tell it that it is permitted to run CGI scripts in a certain directory? We do it with ExecCGI option. Where do we add it? In Fedora, Apache configuration is modular. Instead of adding everything to the main httpd.conf file, and then struggling with it when you upgrade Apache, we add bits of local configuration to files in /etc/httpd/conf.d directory. So, I went ahead, created a new file usercgi.conf in that directory and entered the following directives in it:

<Directory /home/cgiuser/public_html>
  Options +ExecCGI
</Directory>

To my surprise, this didn't work. Apache refused to run CGI scripts in /home/cgiuser/public_html. Giving it more thought, I realized that the files in cond.d directory are included into the main httpd.conf before most of the directives there, and the main httpd.conf has the following part regarding users' public_html directories:

<Directory /home/*/public_html>
    Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
    ...  
</Directory>

The default directive comes after my custom directive and overrides it. What can we do? There are three approaches:
  1. Brute-force approach. Modify the main httpd.conf to include our custom directive after the default directive. I don't like this though, since it's easy to loose the configuration when upgrading to a newer version of Apache.
  2. Put the user's CGI scripts in a subdirectory of public_html, say public_html/cgi-bin. Then you will be able to turn ExecCGI on this directory on with the usual directives:
    
    <Directory /home/cgiuser/public_html/cgi-bin>
      Options +ExecCGI
    </Directory>
    
    
    Why does it work for subdirectory? Because a more specific Directory directive (the one with a longer path) always overrides the one that is less specific. Therefore options set for /home/cgiuser/public_html/cgi-bin will override those set for /home/*/public_html.
  3. Use Location directive instead of Directory:
    
    <Location ~/cgiuser/>
      Options +ExecCGI
    </Directory>
    
    
    Apache always processes Location directives after all Directory directives, thus a Location directive will take precedence.

Friday, April 04, 2008

NTLM authentication in Apache

NTLM authentication is Microsoft's way to enable Windows users to transparently (i.e. without entering user name and password) authenticate to web sites. Apache can support this style of authentication too with mod_auth_ntlm_winbind. This module is developed as part of Samba. It is available in Fedora. More information on it can be found here: http://adldap.sourceforge.net/wiki/doku.php?id=mod_auth_ntlm_winbind, including the basic information on how to install and configure this module.

An important piece of information that I didn't see mentioned by any documentation and that I discovered the hard way is that Apache's KeepAlive directive should be on, otherwise NTLM authentication doesn't work. Unfortunately Fedora 7 default httpd.conf has "KeepAlive off" (don't know why). So, one has to turn it into "KeepAlive on" to use mod_auth_ntlm_winbind. With keep-alives turned on, it works great.

Monday, March 17, 2008

Еще правила для SpamAssassin'а

В продолжение темы спама, вот модификация правил из предыдущей статьи, которая по-прежнему актуальна и эффективна:


header __SPAMMY_YAHOO_SENDER From:addr =~ /[A-Z][a-z]*[A-Z][a-z]*[0-9][0-9]\@yahoo\.co\.uk/
header __SUBJECT_SEMINAR Subject =~ /seminar|Moratoriy/
header __SUBJECT_STARTS_WITH_DOT Subject:raw =~ /^\s*=\?Windows-1251\?Q\?\._?=/

meta SPAMMY_YAHOO_SENDER_AND_SUBJECT __SPAMMY_YAHOO_SENDER && (__SUBJECT_SEMINAR|| __SUBJECT_STARTS_WITH_DOT)
describe SPAMMY_YAHOO_SENDER_AND_SUBJECT Spammy sender FirstLastNN@yahoo.co.uk and suspicious subject
score SPAMMY_YAHOO_SENDER_AND_SUBJECT 2.5

Здесь должно быть 6 строк, начинающихся словами header, header, header, meta, describe и score. Любые другие переносы строк надо удалить!

Что здесь происходит: мы отслеживаем письма, у которых адрес отправителя имеет вид FirstLastNN@yahoo.co.uk (где First и Last - произвольные имя и фамилия, а NN - двузначное число), а тема письма - либо слова seminar или Moratoriy, либо начинаются на точку. Сомневаюсь, чтобы кому-то кроме спаммеров пришло в голову делать такую тему письма, так что такие письма наверняка спам.

P.S. 23.04.2008. Теперь такой же спам присылают также с адресов вида *@earthlink.net. Ввиду этого я модифицировал правила таким образом:


header __SPAMMY_YAHOO_SENDER From:addr =~ /[A-Z][a-z]*[A-Z][a-z]*[0-9][0-9]\@yahoo\.co\.uk/
header __SPAMMY_EARTHLINK_SENDER From:addr =~ /\@earthlink\.net/
header __SUBJECT_SEMINAR Subject =~ /seminar|Moratoriy/
header __SUBJECT_STARTS_WITH_DOT Subject:raw =~ /^\s*=\?Windows-1251\?Q\?\._?=/

meta SPAMMY_SENDER_AND_SUBJECT (__SPAMMY_YAHOO_SENDER || __SPAMMY_EARTHLINK_SENDER) && (__SUBJECT_SEMINAR|| __SUBJECT_STARTS_WITH_DOT)
describe SPAMMY_SENDER_AND_SUBJECT Spammy sender and suspicious subject
score SPAMMY_SENDER_AND_SUBJECT 2.5

Tuesday, December 25, 2007

Еще одно правило для SpamAssassin'a

В продолжение темы предыдущей статьи. Еще один "популярный" в последние дни спам предлагает базы с частной информацией. Subject письма произвольный, и приходят эти письма с произвольных адресов. Пока все эти адреса не попали в блок-листы, SpamAssassin пропускает эти письма. К счастью, спаммеры снабдили свои письма весьма необычным заголовком X-Mailer:

X-Mailer: Sendmail 8.12.11/8.12.11

Прокол со стороны спаммеров! Благодаря такому уникальному заголовку, этот спам легко отсеять с помощью следующих правил для SpamAssassin'a:


header MAILER_SENDMAIL X-Mailer =~ /^Sendmail 8\.12\.11\/8\.12\.11/
describe MAILER_SENDMAIL Spammy X-Mailer header
score MAILER_SENDMAIL 2.5

P.S. 26.12.2007. Похоже, отправители этих писем уже попали в блок-лист SpamCop, и теперь в дополнительном правиле нет необходимости.

Friday, December 21, 2007

Спам на букву "i"

Последнее время мы часто получаем новую разновидность спама. Это сообщения на русском языке на такие темы как "защита конфиденциальной информации предприятия", "защита коммерческой тайны", "Тренинг по продажам" и проч., причем доставляются они как будто через легальный сервер Yahoo. Не знаю как спаммеры это делают, но в результате SpamAssassin пропускает эти сообщения, т.к. единственный тест, который на них срабатывает - BAYES_99. Хоть это и очень важный тест, все равно его одного не достаточно. (Иногда возникает желание поднять score этого теста до 5.4, как это было в SpamAssassin'e версии 2, но я все же думаю, разработчики выбрали score 3.5 не зря - false positives никому не нужны).

К счастью, этот спам можно отсеять благодаря его странной особенности - subject всех этих сообщений начинается с латинской буквы i. В нераскодированном виде начало subject'а всегда выглядит так:

=?Windows-1251?Q?i_=

Никто в здравом уме не будет делать такой subject. Поэтому достаточно добавить в конфигурационный файл SpamAssassin'a (обычно /etc/mail/spamassassin/local.cf) следующие строки (здесь должно быть 3 строки, начинающихся со слов header, describe и score, но ваш браузер может их разбить в неудачном месте):


header SUBJECT_STARTS_WITH_I Subject:raw =~ /^ *=\?Windows-1251\?Q\?i_=/
describe SUBJECT_STARTS_WITH_I Subject has a common spam pattern
score SUBJECT_STARTS_WITH_I 2.5

и спам на букву "i" будет успешно отфильтрован.

P.S. 26.12.2007. Похоже, отправитель этих сообщений уже попал в блок-лист SpamCop, и в дополнительном правиле теперь нет необходимости.

P.S. 06.02.2008. Этот тип спама возвращается, причем теперь subject может начинаться не только на i, но и на точку. Обновленное правило:


header SUBJECT_STARTS_WITH_I Subject:raw =~ /^\s*=\?Windows-1251\?Q\?[i\.]_=/
describe SUBJECT_STARTS_WITH_I Subject has a common spam pattern
score SUBJECT_STARTS_WITH_I 2.5