README.TXT

INTRODUCTION
------------

Carbonite is a tool used to process the /proc filesystem information
in a reliable fashion.  It is built upon the work that Dominique Brezinski
started with his cryogenic program.  Cryogenic is a 'user space' forensic
tool to recover live data and process the /proc filesystem on Linux
machines.  However, current 'rootkit' loadable kernel modules (such as 
knark and heroin) permit system calls to be intercepted, preventing
cryogenic and ps from processing such hidden executables.  

Therefore we created Carbonite, a cryogenic, lsof and ps at the kernel level.  
It queries every process in Linux's task_struct, which is the kernel 
structure that maintains information on every running process in Linux.  
There is little doubt in our minds that attacker's will read this code, 
and find another way to *hide* their processes.  This module will at 
least raise the bar a and provide system administrators with a more reliable 
method to identify all running processes on the system.

CARBONITE FEATURES
------------------
-  Provides nice output for findings in ps like fashion in CARBONITE.txt.
-  Reviews each process for 'out of bounds' process flags (as used by
   knark, heroin, and other rootkit LKMs to tag a process as invisible)
-  Creates an exe file for each running process (whether linked or unlinked).
-  Checks processes for open files, sockets, and pipes and lists them in 
   CARBONITE.txt.
-  Reports the inode number for each open file descriptor, whether it is a
   socket, file, or interprocess pipe.
-  We have included a carbonite.html.c for folks who wish to have a version
   that creates an html output file.

Drawbacks:
-  Writes information on 'evidence' hard drive.
-  Only currently available for the 2.2 Linux Kernel.
-  Reports start time in jiffies since start up rather than real time.
   - Not much of a drawback as you can still identify processes which 
     started up during the boot process.

USAGE
-----
Here are the simple instructions:
-  make
-  mkdir /tmp/CARBONITE
-  ./carbonite.sh

Here is the more in-depth version:
-  Change the 'valid_flags' to reflect the target kernel compilation. The 
   current valid flags can be viewed in /usr/src/linux/include/linux/sched.h  
   The default setting for carbonite.c should work in most cases.
-  Change the PF_TOTALNUM definition to equal the number of valid flags in 
   'valid_flags'.
-  Compile the code using the included makefile by typing 'make'.  If you are 
   compiling this for a SMP (more than one processor) kernel, then you must 
   uncomment the 'CFLAGS' line in the makefile.  This must be done even if you 
   have only one processor and you are still running a SMP kernel.
-  Use the provided shell script 'carbonite.sh' to: 
	-  delete any files in /tmp/CARBONITE
	-  mkdir /tmp/CARBONITE
	-  insmod carbonite.o
	-  rmmod carbonite.o
-  By default, /tmp/CARBONITE will contain dynamically compiled versions of all 
   the running processes in the following format:
	/tmp/CARBONITE/CARBONITE.<command>.PID
-  Carbonite will create a /tmp/CARBONITE/CARBONITE.txt file that has 'ps' and 
   lsof like output of running processes.
-  You may want to edit carbonite.sh to reflect any changes such as directory 
   variations within carbonite.c

SAMPLE OUTPUT FILE
------------------

CURRENT LOCAL TIME: 03-20-01 17:20:42

PID	H?  UID	STAT		COMM	START TIME (Jiffies)
1751	N     0	R (running)	insmod	0 Days ago at 17:20:42 (3281641)
0	N     0	R (running)	swapper	0 Days ago at 08:13:46 (0)
1	N     0	S (sleeping)	init	0 Days ago at 08:13:46 (33)
2	N     0	S (sleeping)	kflushd	0 Days ago at 08:13:46 (33)
3	N     0	S (sleeping)	kupdate	0 Days ago at 08:13:46 (33)
4	N     0	S (sleeping)	kpiod	0 Days ago at 08:13:46 (33)
5	N     0	R (running)	kswapd	0 Days ago at 08:13:46 (33)
6	N     0	S (sleeping)	mdrecoveryd	0 Days ago at 08:13:52 (606)
279	N     1	S (sleeping)	portmap	0 Days ago at 08:14:08 (2250)
294	N     0	S (sleeping)	lockd	0 Days ago at 08:14:08 (2256)
295	N     0	S (sleeping)	rpciod	0 Days ago at 08:14:08 (2256)
304	N     0	S (sleeping)	rpc.statd	0 Days ago at 08:14:08 (2259)
318	N     0	S (sleeping)	apmd	0 Days ago at 08:14:08 (2295)
369	N     0	S (sleeping)	syslogd	0 Days ago at 08:14:09 (2321)
378	N     0	R (running)	klogd	0 Days ago at 08:14:09 (2334)
392	N    99	S (sleeping)	identd	0 Days ago at 08:14:09 (2392)
395	N    99	S (sleeping)	identd	0 Days ago at 08:14:10 (2407)
396	N    99	S (sleeping)	identd	0 Days ago at 08:14:10 (2407)
398	N    99	S (sleeping)	identd	0 Days ago at 08:14:10 (2409)
399	N    99	S (sleeping)	identd	0 Days ago at 08:14:10 (2409)
410	N     0	S (sleeping)	atd	0 Days ago at 08:14:10 (2422)
424	N     0	S (sleeping)	crond	0 Days ago at 08:14:10 (2437)
439	N     0	S (sleeping)	cardmgr	0 Days ago at 08:14:11 (2535)
453	N     0	S (sleeping)	lpd	0 Days ago at 08:14:11 (2564)
505	N     0	S (sleeping)	sendmail	0 Days ago at 08:14:13 (2742)
542	N     0	S (sleeping)	pump	0 Days ago at 08:14:13 (2764)
545	N     0	R (running)	gpm	0 Days ago at 08:14:13 (2772)
584	N    43	S (sleeping)	xfs	0 Days ago at 08:14:14 (2870)
619	N     0	S (sleeping)	login	0 Days ago at 08:14:16 (3037)
620	N     0	S (sleeping)	mingetty	0 Days ago at 08:14:16 (3037)
621	N     0	S (sleeping)	mingetty	0 Days ago at 08:14:16 (3037)
622	N     0	S (sleeping)	mingetty	0 Days ago at 08:14:16 (3037)
623	N     0	S (sleeping)	mingetty	0 Days ago at 08:14:16 (3037)
624	N     0	S (sleeping)	mingetty	0 Days ago at 08:14:16 (3037)
657	N   500	S (sleeping)	csh	0 Days ago at 08:15:30 (10461)
680	N   500	S (sleeping)	xinit	0 Days ago at 08:15:35 (10960)
681	N   500	R (running)	X	0 Days ago at 08:15:35 (10991)
684	N   500	S (sleeping)	xterm	0 Days ago at 08:15:38 (11249)
686	N   500	S (sleeping)	csh	0 Days ago at 08:15:39 (11313)
687	N   500	R (running)	kwm	0 Days ago at 08:15:44 (11832)
708	N   500	S (sleeping)	kbgndwm	0 Days ago at 08:15:48 (12227)
715	N   500	R (running)	kfm	0 Days ago at 08:15:48 (12257)
716	N   500	S (sleeping)	krootwm	0 Days ago at 08:15:48 (12257)
717	N   500	S (sleeping)	kpanel	0 Days ago at 08:15:48 (12257)
724	N   500	S (sleeping)	csh	0 Days ago at 08:15:51 (12534)
726	N   500	S (sleeping)	xterm	0 Days ago at 08:15:51 (12536)
732	N   500	S (sleeping)	csh	0 Days ago at 08:15:51 (12554)
902	N   500	S (sleeping)	su	0 Days ago at 09:25:11 (428547)
904	N     0	S (sleeping)	bash	0 Days ago at 09:25:18 (429236)
914	N     0	S (sleeping)	csh	0 Days ago at 09:25:21 (429521)
1188	N   500	R (running)	netscape-commun	0 Days ago at 13:22:45 (1853916)
1219	N   500	S (sleeping)	netscape-commun	0 Days ago at 13:22:45 (1853932)
1615	N   500	S (sleeping)	su	0 Days ago at 16:57:01 (3139541)
1617	N     0	S (sleeping)	bash	0 Days ago at 16:57:12 (3140644)
1629	N     0	S (sleeping)	csh	0 Days ago at 16:57:26 (3142083)
1679	N   500	S (sleeping)	konsole	0 Days ago at 17:08:45 (3209957)
1680	N   500	S (sleeping)	csh	0 Days ago at 17:08:45 (3209998)
1682	N   500	S (sleeping)	su	0 Days ago at 17:09:15 (3212963)
1684	N     0	S (sleeping)	bash	0 Days ago at 17:09:21 (3213563)
1695	Y     0	S (sleeping)	xterm	0 Days ago at 17:09:34 (3214855)
1697	N     0	S (sleeping)	bash	0 Days ago at 17:09:34 (3214884)
1748	N     0	S (sleeping)	sh	0 Days ago at 17:20:42 (3281639)

/* For Each PID you will also get the following report on file descriptors open
 and environment variables*/

****
PID: 1697     
COMMAND: bash                
ARGS: bash

ENV: LESSOPEN=|/usr/bin/lesspipe.sh %s
COLORTERM=
HOSTNAME=mikebarry
LOGNAME=mbarry
MAIL=/var/spool/mail/mbarry
MACHTYPE=i386
TERM=xterm
HOSTTYPE=i386-linux
PATH=/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin
KDEDIR=/usr
HOME=/root
INPUTRC=/etc/inputrc
SHELL=/bin/bash
USER=mbarry
VENDOR=intel
GROUP=mbarry
QTDIR=/usr/lib/qt-2.1.0
DISPLAY=:0.0
LANG=en_US
HOST=mikebarry
OSTYPE=linux
WINDOWID=50331662
PWD=/home/mbarry
SHLVL=4
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
_=/usr/X11R6/bin/xterm

FILES: 
0:     /3
inode number: 5         
1:     /3
inode number: 5         
2:     /3
inode number: 5         
****

****
PID: 1733     
COMMAND: sh                  
ARGS: /bin/sh
./carbonite.sh

ENV: LESSOPEN=|/usr/bin/lesspipe.sh %s
HOSTNAME=mikebarry
LOGNAME=mbarry
MAIL=/var/spool/mail/mbarry
MACHTYPE=i386
TERM=xterm
HOSTTYPE=i386-linux
PATH=/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin
KDEDIR=/usr
HOME=/root
INPUTRC=/etc/inputrc
SHELL=/bin/bash
USER=mbarry
VENDOR=intel
GROUP=mbarry
QTDIR=/usr/lib/qt-2.1.0
DISPLAY=:0.0
LANG=en_US
HOST=mikebarry
OSTYPE=linux
WINDOWID=37748750
PWD=/home/mbarry/carbonite
SHLVL=6
LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
_=/bin/csh

FILES: 
0:     /2
inode number: 4         
1:     /2
inode number: 4         
2:     /2
inode number: 4         
3:     /home/mbarry/carbonite/carbonite.sh
inode number: 66069     
****




During Incident Response:

-  Remove any printk lines in init-module and cleanup-module so no messages are 
   handled by the syslogd. Why alert the intruder(s)?
-  Rename the module to something insignificant other than carbonite.
-  Change the file 'prepends' to reflect any changes to this name.  This will 
   prevent simple searching on the intruder's part for the name of carbonite.
-  Compile carbonite on a trusted, reliable system with the same kernel version
   as the victim system.

HISTORY (Why we did this)
-------------------------
With the advent of LKM's that intercept the getdents system call, the kill 
system call, and any other system calls, a typical system administrator who uses
'user space' tools to query running process could lose critical information in 
an incident response. For an example, if the getdents syscall is intercepted, 
the /proc information you review will be unreliable. 

Creed wrote a LKM called 'knark' which hides running processes by issuing a 
'kill -31' (can be modified) to a desired PID.  The process with PID will not 
appear in the /proc filesystem or in the PS output, because PS merely queries 
/proc directories.  

Knark hides a process by assigning a (default) flag named PF_INVISIBLE of 
0x10000000.  This is not a valid process flag. The process flag is a 32 bit 
unsigned integer, and valid values are defined in sched.h as:

/***********************************/

struct task_struct {
/* these are hardcoded - don't touch */
	volatile long state;	/* -1 unrunnable, 0 runnable, >0 stopped */
	unsigned long flags;	/* per process flags, defined below */
	int sigpending;
	mm_segment_t addr_limit;	/* thread address space:
					 	0-0xBFFFFFFF for user-thead
						0-0xFFFFFFFF for kernel-thread
					 */
	struct exec_domain *exec_domain;
	long need_resched;

<skipped a lot>

/*
 * Per process flags
 */
#define PF_ALIGNWARN	0x00000001	/* Print alignment warning msgs */
					/* Not implemented yet, only for 486*/
#define PF_STARTING	0x00000002	/* being created */
#define PF_EXITING	0x00000004	/* getting shut down */
#define PF_PTRACED	0x00000010	/* set if ptrace (0) has been called */
#define PF_TRACESYS	0x00000020	/* tracing system calls */
#define PF_FORKNOEXEC	0x00000040	/* forked but didn't exec */
#define PF_SUPERPRIV	0x00000100	/* used super-user privileges */
#define PF_DUMPCORE	0x00000200	/* dumped core */
#define PF_SIGNALED	0x00000400	/* killed by a signal */
#define PF_MEMALLOC	0x00000800	/* Allocating memory */
#define PF_VFORK	0x00001000	/* Wake up parent in mm_release */

#define PF_USEDFPU	0x00100000	/* task used FPU this quantum (SMP) */
#define PF_DTRACE	0x00200000	/* delayed trace (used on m68k, i386) */

/*********************************************/

Looking at this you should notice that there are 19 unused bits, and who knows 
how many combination bits that can be used to flag a process as invisible. In 
fact, we have noticed many areas in the task_struct that are currently unused, 
and each one can be used as a flag to hide a process.


SOME TECHNICAL DETAILS
----------------------

We do not want to release some code and not explain what it does.  We hope you 
take this program further and share your ideas. 

Every active Linux process occupies exactly one entry in the process table.  We 
use the iterate through each task_struct, the structure that maintains all the 
information for one running process, by grabbing the currently running process 
using the 'current' global variable.  The task_struct is part of a linked list 
so each process can be viewed.  Furthermore, the linked list is in a circular 
fashion where the last process links to the first.  Each task_struct includes 
the following information about a process:
-  task state
-  task per process flags
-  thread address space
-  process credentials
-  file system info
-  ipc information
-  memory management information
-  signal handlers
-  thread group tracking

and much more a kernel needs to run a process.

The Carbonite Process:
-  Carbonite opens a file (defaults to /tmp/CARBONITE/CARBONITE.txt) 
-  Carbonite iterates through each process in task_struct and writes the
   process information to the CARBONITE.txt file.
-  Carbonite opens, writes, then closes an executable image file by reading the 
   virtual memory's inodes that are deemed executable.  It does this by
   checking the virtual memory page's flag appropriately.
-  Carbonite then lists any arguments, environment variables, open files, 
   pipes, or sockets in the CARBONITE.txt file by iterating through the 
   process list once again.

Some More Technical Notes:
-  This kernel module will not work on multi-processor enabled kernels without 
   the -D__SMP__ compile option.
-  This kernel module will not work on single-processor processor enabled 
   kernels with the D__SMP__ compile option.
-  PF_ALIGNWARN is not implemented in sched.h
-  If it is not logically acceptable to have combinations of known 
   valid process flags enabled, carbonite will not considered them as "HIDDEN". 
   Currently, carbonite only considers out of range process flags as signals 
   of a process being hidden.
-  Several kernel functions were implemented using Linux kernel source code and
   included in extra source files.

AUTHORS
-------
Kevin Mandia - Director of Computer Forensics, Foundstone, Inc.
Email: kevin.mandia@foundstone.com

Keith J. Jones - Principal Forensics Consultant, Foundstone, Inc. 
Email: keith.jones@foundstone.com

A special thanks to Mike Barry of Foundstone for a tremendous job in testing the code and making special improvements.

LICENSE
-------

All source code released with carbonite is licensed under the GNU license.  
Foundstone, Inc. offers no warranties, explicit or otherwise for damages from 
use or non use of this tool or derivations thereof.
