Feature Proposal: Support multiple password encodings for existing users.

Motivation

We have no easy way to migrate between password encodings, and our current default of crypt is not very secure. Currently to change this is very disruptive. The information needed to make this more flexible is already embedded in the .htpasswd file but we don't make use of it.

Description and Documentation

Add code to HtPasswdUser.pm to understand the "magic" stored in the existing .htpasswd file password hash. $Foswiki::cfg{Htpasswd}{Encoding} changes to be used only when writing a new password. Existing passwords are checked using the embedded magic from .htpasswd.

Encoding	Detected	Example
Digest (md5)	3 colon delimited field, 32 character hash	admin:The Realm:11fbe079ed3476f7712030d24042ca35
SHA-1	{SHA} magic in hash, 33 characters total	admin:{SHA}QvQHx34cyGz2cjXj6cauQoAwtIg=
Crypt	(no magic) - 11 character hash	admin:$cnhJ7swqUWTc
Apache MD5	$apr1$ magic in hash<br />(Not supported by Foswiki)	admin:$apr1$jgwedrkq$jzeetEHMGal5H0SUFDMEl1
crypt-MD5	$1$ magic in hash, 34 characters total	admin:$1$3iuE5z/b$JHyXMzQOIq3cl6WlEMoZC.

Foswiki supports and can generate all of these formats except for the Apache MD5 with $apr1$ magic.<

This enhancement request does the following:

• Add an extra field to the return from HtPasswdUser::fetchPass(). It will have the "database" entry for the existing password as recovered from .htpasswd. Added to the entry include the "Encoding" decoded from the magic, and the old Realm for md5 digest passwords.
  • Only returned if the caller requests an array.
• Modify =HtPasswdUser::encrypt() to support an optional field - the "database" entry recovered from the existing password. If provided, encrypt will use the old Encoding and Realm to encode the new password, allowing comparison of the passwords.
  • Default to configured $Foswiki::cfg{Htpasswd}{Encoding} and {AuthRealm} if not provided.
• Modify HtPasswdUser::checkPasswd() to use the password entry returned by fetchPass to override the encryption type.
• Add the "real" apache $apr1$ routine. See password_encryptions which defines this as: "$apr1$" + the result of an Apache-specific algorithm using an iterated (1,000 times) MD5 digest of various combinations of a random 32-bit salt and the password. This adds another optional CPAN dependency: Crypt::PasswordMD5. It is only needed if apache-md5 is selected, or AutoDetect is enabled.

The big advantage here is that the default {Htpasswd}{Encoding} can be changed without invalidating everyone's passwords. Encoding is updated user by user as each password is reset or changed. If the admin desires a mass change, the bulk password reset can still be used.

In testing this, it appears as though Apache mod_auth works this way already. HTTPD auth login works fine with mixed encodings in the .htpasswd file.

There is also a documentation issue - probably not worth changing, as it would be disruptive, but:

• our md5 encoding is actually equivalent to the Apache "Digest" authentication generated with htdigest.
• our crypt-md5 has no equivalent in apache docs.
• Apache MD5 generated with htpasswd -m option has no equivalent in Foswiki.

It would be easier for admins if our terminology aligned better with Apache terminology.

I've addressed this by adding a checker and guess code to change md5 to htdigest-md5 hopefully eliminating some confusion.

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: GeorgeClark - 13 Jul 2011

Discussion

SvenDowideit pointed out a number of considerations on IRC:

• File is "csv" - colon separated. So it is possible that sites might have added fields beyond our email extension. Counting colons might not be reliable.
• Also email address might be missing from a digest entry, which would possibly mis-identify a digest entry.
• Also if Foswiki doesn't manage the htpasswd file, then email entries will not be present.
• Auto-detect might be considerable overhead. The entire .htpasswd file is loaded on every transaction for non-persistent-perl sites. A few milliseconds per user will add up. So Auto-detect might need to be configurable.
• Might be worthwhile having a config checker that warns if there are mixed encodings in the file.

-- GeorgeClark - 13 Jul 2011

It doesn't look like there is a huge performance hit going to automatic recognition of the password encoding. Running 4 versions of the code against a .htpasswd file with 4400 entries, the following timings are generated:

Implementation	Seconds	Delta (milliseconds)
trunk non-auto	0.035712	-16.305
trunk Auto Detect	0.039323	-12.694
1.0.9 md5 digest	0.052017	(baseline)
1.0.9 crypt	0.059067	+7.05

The performance improvement was due to changing from regex parsing to split. So using Auto-detect has about a 4ms hit, but it's still 12ms faster than 1.0.9 for 4400 entries in .htpasswd.

It turns out it's also really simple now to use the "old realm" stored in .htpasswd when checking the password, and using the configured realm when writing a new password. This allows the {AuthRealm} setting to be changed without invalidating current passwords, at least when using Template auth.

Is there a reason that we want to invalidate all passwords with a changed realm, or is allowing migration okay.

-- GeorgeClark - 15 Jul 2011

Sadly I've not had an opportunity to use auth realms in practice. However, I'm really looking forward to this feature

-- PaulHarvey - 16 Jul 2011

I've also deprecated the md5 setting, and added checker code to change it to htdigest-md5. The code accepts either value, and added unit test for both cases. The help text has been changed to hopefully be more useful without requiring the supplemental document.

The choices in order of strongest to lowest strength:

HTTPS

Any below encoding over an HTTPS SSL connection. (Not a selection here.)

htdigest

Strongest only when combined with the Foswiki::LoginManager::ApacheLogin Useful on sites where password files are required to be portable. The {AuthRealm} value is used with the username and password to generate the encrypted form of the password, thus: user:{AuthRealm}:hash. This encoding is generated by the Apache htdigest command.

sha1

is recommended. It has the strongest hash. This is the encoding generated by the htpasswd -s command (userid:{SHA}hash).

apache-md5

Enable an Apache-specific algorithm using an iterated (1,000 times) MD5 digest of various combinations of a random 32-bit salt and the password (userid:$apr1$salt$hash). This is the encoding generated by the htpasswd -m command.

crypt-md5

Enable use of standard libc (/etc/shadow) crypt-md5 password (like user:$1$salt$hash:email). Unlike crypt encoding, it does not suffer from password truncation. Passwords are salted, and the salt is stored in the encrypted password string as in normal crypt passwords. This encoding is understood by Apache but cannot be generated by the htpasswd command.

crypt

is the default. Not Recommended. crypt encoding only uses the first 8 characters of the password. Extra characters are silently discarded. This is the default generated by the Apache htpasswd command (user:hash:email)

plain

stores passwords as plain text (no encryption). Useful for testing. Not compatible with {AutoDetect} option.

If you need to create entries in .htpasswd before Foswiki is operational, you can use the htpasswd or htdigest Apache program to create a new password file with the correct encoding. Use caution however as these programs do not support the email addresses stored by Foswiki in the .htpasswd file.

---

Excellent work George! Can we make a 1.2/2.0 release blocker to make the default, something that won't cause a warning? I suppose we could just do it on trunk under Tasks.Item10962, if we can do it before 1.1.4 is released.

-- PaulHarvey - 10 Aug 2011