Nerdpedes,
have a look at
https://security.stackexchange.com/questions/64915/what-are-the-biggest-security-concerns-on-pxe
Best of:
I can capture a full machine image. Do your systems automatically connect to the domain controller after setting up the machine? If so, this image probably has domain controller credentials on it, that I can capture and use elsewhere.
Computer makes a DHCP request --> DHCP server responds with address and PXE parameters --> Computer downloads boot image using TFTP over UDP
If the good guys got the traffic on that low level, unencrypted. Then they would have it all.
In over simplified terms, It allows you to ignore the onboard Operating configuration and instead of booting from the internal drive, it downloads another configuration or additional executables from over the network.
So a machine that has been audited can then have PXE enabled post audit to boot up an entirely different image or functions than which was audited.
However, in this scenario they would have to fake the logs on the built in OS for the required dates, otherwise it would be obvious it wasn't in use at the relevant time.
good point
Wasn't there something about missing/deleted logs a bit back? Maybe they were not deleted, but the logs are on a remote profile somewhere and would account for missing log entries? I am pretty ignorant of PXE boot so if the above is stupid feel free to call me out and correct please.
I would guess that if they were network booting the voting machines then the missing logs are part of the evidence the audits picked up.
The story I heard was that the logs are a finite size and someone had logged in well after the election and had caused many records to be written to the log. That caused the records from the election being "scrolled" off the top and lost.
Yes, to expand on what you've said.
IDRAC allows both remote control, and BIOS configuration changes. With the BIOS changes they could be configured to boot from network first essentially having them boot "Dirty" via PXE Boot during the elections. The dirty OS run its own version of the vote counting software that the auditors never see, and writes the results to the Database on Disk. Once the elections are over. They boot clean from disk. When an auditor powers them back on the PXE server isnt on the network so the boot they disk, and load the clean OS and all the clean programs, and appear good.
In sprit this has a lot of parallels to how Volkswagen cheated, albeit more technical and with a few more steps, but at the end of the day these ran DIRTY during the election, and would appear CLEAN to an auditor or anyone who powered them up.
The thing is about all this is the router logs would have a good chance of catching this via logs on DHCP forwarding, which is why the Maricopa Country Board of Supervisors is fighting tooth and nail not to turn over the logs.
Up up
And after the machine shuts down, the entire partition that the PXE was mounted on can be reformatted, and written over with military grade erasure so there is no evidence that the PXE boot was invoked
This is how computer builders load the image on your PC or Laptop and configure it the way you ordered
PXE can be used to silently and automatically install a modified, or entirely different operating system, including one which runs as an ISO (CD/DVD) image, without actually modifying the hard drive, leaving no trace of it, when machine is rebooted.
Moreover, it allows one to change out the operating system image from any other point on the network.
Are you talking about the winPE environment, I've never seen the ability to download and run an iso in ram without touching the HDD like a live cd. Can you please provide info or instruction on this?
TAILS linux.
Yes but we are talking about pxe booting tails then, I've never seen that done.
I haven't either, I just meant an OS from an ISO image. I suppose you could put the image on a disk and serv that from PXE or something like it.
It would run all in RAM and not touch the HD. That's sort of like an old WYSE dumb terminal.
I see what you are saying, I just don't know if it's possible to 'stream' an OS like that. Modern OSs are generally large and complex. The way this would work could be potentially a thin client or similar, but then we are talking about terminal services rather than PXE.
WinPE can run from RAM, you could boot to a virtual disk over the network, most linux flavors can run as a Live CD...
You can get a lot of stuff running in a WinPE environment, especially if you're the one writing the software.
Yes, this is all true but you need to exit the winPE environment to run the OS. You say you can boot to a virtual disk over the network. Wouldn't this be a thin client / terminal services rather than PXE?
WindowsPE runs in RAM I boot to it from a flash drive and pull out the drive once it’s up all the time
Yes, but have you pxe booted a live cd or similar, that's what we are talking about here. The pxe environment is meant for installing on hdd not running live, I don't know that it's possible.
ah, indeed. So....this is from microsoft. https://docs.microsoft.com/en-us/troubleshoot/mem/configmgr/understand-pxe-boot
"The NBP downloads the operating system loader and the boot files via TFTP, which include the following:
smsboot\x64\pxeboot.com smsboot\x64\bootmgr.exe \SMSBoot\Fonts\wgl4_boot.ttf \SMSBoot\boot.sdi \SMSImages\RR200004\boot.RR200004.wim A RAMDISK is created using these files and the WinPE WIM file in memory.
RAMDISK is created
The client boots from the RAMDISK."
Looks like pxe does boot from RAM? I was under the impression that PXE was for deploying an image/images out easily to clients for installation of said images onto the clients drive. I am not certain anymore due to reading through the doc I linked from microsoft.
OK but where is it saving the .wim file? These files are often large and far too big to simply store it in ram, so, when it downloads, where does it write it, if not the hdd?
I don't know, but a windows LTSC .wim is 3GB, I wouldn't say that its to big for modern RAM. I haven't come across anything under 8GB RAM in a while
I'm not really arguing with you, you are correct, it's possibleto fit it all in ram, just not likely. That said the image for the voting machines is probably pretty stripped down and much smaller than a standard windowsimage. I have similar at work, but our image is still in the 40-50 gig range. .wim stands for windows image file. Usually it is a presetup windows machine at 20+ gigs. From my experience. I service machines for a company that use pxe for deployment and will often image many machines at once. Not saying you are wrong tho.
Could it be where the infamous thumb drives come into play? to serve up the required software overrides across a LAN?
It's possible, yes.
TY - That's similar to what I was about to say.
My question in this realm is are SSD or other flash media drives forensically retraceable once a new OS is reinstalled?
It depends. Data written to a flash or SSD drive, before either deleting the file and/or reformattng can often be recovered -- easily so, if that region of the drive has not yet been rewritten over with other data. If it has, there are forensic IT tools which can still sometimes recover the old data using statistical analysis and other techniques.
However, an OS which is loaded via PXE, and running in RAM only would leave very little evidence behind, assuming all operations it performed were also in RAM only, once the machine were rebooted (clearing anything in RAM). Analyzing what was previously only in RAM is also possible, but requires more expertise, and favorable conditions.
Also worth mentioning, is that some newer computers and devices often have a built-in 'side computer' which can be running and even accessed remotely, even when the main computer is turned off. The marketing reason for this is to enable remote administration/IT assistance, even when a machine has crashed or is not booted nor powered on.
Thank you for the great info. to your last statement, I know some HP enterprise servers have a separate NIC and rom for remote built in.
HP, Dell, IBM. They all have that. DRAC or ILO, each their own version of the same thing. Access to power cycle, BIOS, and to the OS if it is running.
IME
The only way to prevent it is to enable secure boot with a signed kernel. If that is unavailable or disabled, a person can boot anything they want without a trace.
Just pointing out that you can run bad software on a legit, signed kernel
The old diebold machines would run anything with the right file name on an sd card. No checks what so ever.
If we have captured traffic showing the PXE communications packets, that would be huge. Otherwise, we just have a 'potential vulnerability' that may or may not have been exploited.
If PXE was used during the election, I believe the built-in OS logs would at least show that since they would have no entries during that timeframe.
PXE operates over Bootp/DHCP routers are often configured as BOOTP/DHCP forwarders. If PXE was turned on their is a chance the router logs would capture it if they were configured as DHCP forwarders.
not just or only a LAN cable, it could be used if a cellular modem or plain wifi card or chip was on-board. I believe this hardware is Chinese manufactured so who knows what may be at chip level.
Not the router logs. Routers log very little to nothing locally, they can however send logs to a server that does store logs which can display slice/dice them. What is important and I wish people would understand, is the router CONFIG file. This has the static routing tables, can show where the pxe images were pulled from.
Also to clarify, for pxe usage, the vast majority of the time, DNS, DHCP, and pxe are not all on the same box, nor should they be, that is really bad practice.
yah, the static routing tables would show the IP of the server where the images would held, and the assumption would be on an internal subnet that the router can route directly to.
PXE is not only used for pushing an image, but for capturing one using Acronis or Ghost. In the pictures from CM, they were using Acronis. Once you capture an image, you have passwords, settings, lots of security data.
I believe they do have it all.
But the enemy has the megaphone and wont give the good guys the time of day to discuss it.
Devolution
The PXE boot plus the remote IDRAC is a terrible combination. . A remote Idrac connection allows them to setup a virtual console, provision a file share over the network as a local disk, or change the BIOS boot order to include a network boot.
A virtual console would allow someone on the network to come in and have keyboard and mouse access to these machines. Database management tools capable of altering the database were found on the machines. This would allow people to come in and basically alter the results the voting machines provided at will.
Provisioning a network share as a local disk. This would allow someone to point to any files they had prepared, and have them accessible. These could log altering to alter the logs on the machines to cover their tracks or any other scenario.
Change the Bios Boot Order to include a network boot. This is perhaps the most subtle and insidious using this could tell the machines to boot and run DIRTY during the election, but after the election they would run the CLEAN version from the disk.
Technical explanation of that 3rd really insidious one. Transfering an OS over the network is data intensive, but there are Minimal OS's designed to run other OS in a containerized fashion. Meaning that instead of booting the CLEAN OS on the disk, they boot a container OS configured to run the OS and Applications on the Disk, but with very subtle variations that the OS would be unaware of. Here is one Scenario
The thing about this is that PXE booting requires DHCP and BOOTP, which have a high probability of hitting the router and being logged by DHCP forwarding configured on the router. Worse yet if the PXE server was configured by someone with access the routers they could simply change the normal forwading address to the rogue server they setup. In this case the evidence would be on the router and network logs at the county, instead of on the Machines. Kindof an Odd coincidence that the county is fighting tooth and nail not to turn these over.
Why are we using “off the shelf” servers for the countries election?
And if the instructions were to disable the idrac nic why do we have have an image of idrac showing an active IP address?