Originally published April 25, 2019 @ 8:12 pm
Sooner or later it will happen: you type something after which you expect a password prompt then, as a reflex, you type the password. However, you fat-fingered the first command, and your password ended up in clear text in your shell history, likely in the system log, and who knows on what remote syslog server.
Logs live forever. I’ve seen servers with logs that were older than the server (probably copied from the old server for some reason). With the help from remote
syslog and various real-time indexers like Splunk or Graylog, as well as NetBackup and such, logs are almost indestructible. Access to log data is generally poorly controlled with
read permission granted to everyone more often than not.
All these reasons, combined with the fact that people make mistakes, turn log files into a valuable source of information for the hackers. It may seem that extracting strings that might be passwords from log data would be very challenging and time-consuming. Not so. Standard requirements for password strength make this task much more manageable.
Here’s a simple example to illustrate this point. Is this string a password:
password? What about this one:
P@ssw0rd1? See my point? Good, let’s continue.
Below is the script (also on GitHub) you can run on just about any modern Linux server to check users’
/var/log/messages for possible passwords. Naturally, you need to be
root to do this. You can modify the script to only look at log files you can access with your credentials.
If you’re a sysadmin and have access to configuration management tools like Salt or Ansible, you can run this script on multiple systems in parallel. Here’s an example of running the script via Salt CLI:
salt "prod-tomcat*" cmd.script "salt://scripts/bash_history_password_find.sh"
And below is sample output. The first three lines look like the user accidentally copy-pasted into command line an encryption key or somesuch. But the other four lines clearly contain two very stupid passwords for which
jjames will receive a beating.
What to do
Here’s what to do if you realize you typed your password in CLI in clear text. It is as simple as one-two-three.
Do not pretend that nothing happened. You made an honest mistake, so don’t exacerbate the situation by making a dishonest one.
Ask yourself these two questions: can I change the password right now and will doing so cause an outage? If the answers most definitely are “yes” and “no”, correspondingly, then change the password and proceed to Step 3
If the answer is anything else, then just proceed to Step 3.
Do not ignore the problem: it will come out sooner rather than later and you’ll surely get in trouble then. Notify your computer security team as soon as possible, or sooner. Notify your supervisor in an urgent manner. Do not email screenshots, log entries, or the actual password.
You may need to submit a trouble ticket, but, once again, do not include screenshots, log entries, or the actual password. In fact, do not commit to electronic record any specifics of the incident: when folks in charge of handling such incidents require more information – they’ll just have to ask you.
Experienced Unix/Linux System Administrator with 20-year background in Systems Analysis, Problem Resolution and Engineering Application Support in a large distributed Unix and Windows server environment. Strong problem determination skills. Good knowledge of networking, remote diagnostic techniques, firewalls and network security. Extensive experience with engineering application and database servers, high-availability systems, high-performance computing clusters, and process automation.