LOPSA-NJ News Aggregator

It is a fact of modern life that you can't unsend email. The problem is that to really unsend email you need a time travel device. It's a shame, really. MS-Exchange has the ability to send a request that will... Tom Limoncelli http://EverythingSysadmin.com

Hi All,

I spent some time last night upgrading the blog to the latest versions of Wordpress and plugins. I also added a mobile version of the site, so handheld device users can actuallly use the site now without zooming in.

Also, with the help of Greg over at Reject Reality, I got WP SuperCache working. This will hopefully let the blog take more and more traffic as time goes on, and delay me ordering a new virtual host (I go with the guys at prgmr.com).

Anyway, the point of this post is to ask you to let me know if you notice anything strange. Drop me an email if you see anything weird, or if the mobile site shows up but shouldn’t, or any of that. This is my blog, and I write in it, but you all are the ones who are good enough to read my stuff. The least I could do is make it easily accessible to you.

Thanks for your patience, and thanks for reading!

PS – I’ve added a Google Translate widget to the far right bar. This is the first step in what I hope will be many that enable this blog to be read by non-english speaking people from around the world. Please let me know if you have any questions or problems with it.

Brendan Scott linked to a couple of articles about CAL (the Copyright Agency Limited) [1]. I have previously written about CAL and the way that they charge organisations for the work of others without their consent [2]. My personal dispute with CAL is that they may be charging people to use my work, I have not given them permission to act on my behalf and will never do so. If they ever bill anyone for my work then it will be an act of piracy. The fact that the government through some bad legislation permitted them to do such things doesn’t prevent it from being piracy – you can’t disagree with this claim without supporting the past actions of China and other countries that have refrained from preventing factories from mass-producing unauthorised copies of software.

The first article concerns the fact that last year CAL paid more than $9,400,000 in salary to it’s employees (including $350,000 to it’s CEO) while it only paid $9,100,000 directly to the authors [3]. It also spent another $300,000 to send it’s executives to a junket in Barbados. It did give $76,000,000 to publishers “on the assumption that a proportion of this money will be returned to authors” – of course said publishers could have used the money to have holidays in Barbados. CAL doesn’t bother to check who ends up with shares of the $76,000,000 so it’s anyone’s guess where it ends up.

The second article is by James Bradley who is an author and director of CAL [4]. He claims that “much” of the $76,000,000 was distributed to authors, although I’m not sure how he would have any idea of how much it was – which is presumably why he used the word “much” instead of some other word with a clearer meaning such as “most“. He also notes that CAL invested $1,000,000 in “projects specifically designed to promote the development and dissemination of Australian writing“, which sounds nice until you consider the fact that none of the authors (apart from presumably the few who sit on the CAL board) had any say in the matter. Can I take a chunk of the $9,400,000 that is paid to CAL employees and invest it in something? If not then why not? If they can “invest” money that was owed to other people then why can’t I invest their salaries?

James also says “The issue of how well CAL serves rights-holders – and authors and artists in particular – is a vital one” which is remarkably silly. He is entirely ignoring the fact that some rights holders don’t want to be “served” by CAL at all. The fact that CAL can arbitrarily take money for other people’s work is an infringement on their rights. He further demonstrates his ignorance by saying “Without CAL and the licences we administer, users – educational institutions, government agencies and corporate organisations, to name just a few – would be required to seek permission every time they reproduced copyright material or run the risk of legal action for copyright infringement” – of course any educational institution can use Creative Commons licensed work [5].

I’ve previously written about the CK12 project to develop CC licensed text books for free use [6]. There’s no reason why the same thing can’t be done for university text books. In the discussion following Claudine Chionh’s LUV talk titled “Humanities computing, Drupal and What I did on my holidays” [7] it was suggested that it should be possible to gain credit towards a post-graduate degree based on work done to share information – this could mean setting up a Drupal site and populating the database or it could mean contributing to CC licensed text books. Let’s face it, a good CC text book will be read by many more people than the typical conference proceedings!

James says that CAL is used “Instead of having to track down individual rights-holders every time they want to reproduce copyright material“. The correct solution to this problem would be to change the copyright law such that if a reasonable attempt to discover the rights-holder fails then work is deemed to be in the public domain. The solution to the problem of tracking down rights-holders is not to deny them their rights entirely and grant CAL the right to sub-license their work!

He also makes the ridiculous claim “Whereas in the age of the physical book schools and universities could have bought fewer books and made up the difference by using photocopies, it is now possible for an organisation to buy a single set of digital materials and reproduce them ad infinitum” which implies that CAL is the only thing saving the profits of authors from unrestricted digital copying. Of course as CAL seems to have no active enforcement mechanisms and they apparently charge a per-student fee they really have no impact on the issue of a single licensed copy being potentially used a million times – extra use apparently won’t provide benefits to the author and use in excess of the licensing scheme won’t be penalised.

He asks the rhetorical question “After all, why go to the expense of creating a textbook (or some form of digital course materials) if you are going to sell only a half-dozen copies to state education departments“. The answer is obvious to anyone who has real-world experience with multiple licensing schemes – you can sell one single copy and make a profit if the price is high enough. The smart thing for the education departments to do would be to pool their resources and pay text book companies for writing CC licensed texts (or releasing previously published texts under the CC). The average author of a text book would probably be very happy to earn $100,000 for their work, the editorial process probably involves a similar amount of work. So if the government was to offer $300,000 for the entire rights to a text book then I’m sure that there would be more than a few publishers tendering for the contract.

According to the CIA World Fact Book there are 2,871,482 people in Australia aged 0-14 [8], that means about 205,000 per year level. CAL charges $16 for each primary and secondary student so the government is paying about $3,280,000 every year per year level. Even in year 12 the number of text books used is probably not more than 10, so it seems to me that if all the money paid to CAL by schools in a single year was instead used to fund Creative Commons licensed text books then the majority of the school system would be covered! The universities have a much wider range of text books but they also have higher CAL fees of $40 per student. After cutting off the waste of taxpayer money on CAL fees for schools that money could be invested in the production of CC licensed university text books.

• [1] http://brendanscott.wordpress.com/2010/03/10/james-bradley-responds-on-slattery-cal-article/
• [2] http://etbe.coker.com.au/2009/09/07/bad-but-appealing-copyright-arguments/
• [3] http://www.theaustralian.com.au/news/arts/copyright-staff-get-more-than-they-give-to-authors-and-artists/story-e6frg8n6-1225831556653
• [4] http://www.theaustralian.com.au/higher-education/opinion-analysis/licensing-body-copyright-agency-limited-has-lasting-value/story-e6frgcko-1225838858623
• [5] http://creativecommons.org/
• [6] http://etbe.coker.com.au/2009/08/29/free-k-12-text-books/
• [7] http://luv.asn.au/2010/03/02
• [8] https://www.cia.gov/library/publications/the-world-factbook/geos/as.html

I’ve got another column up at Simple Talk: Sysadmin today. It’s called Unteachable Disaster Recovery Techniques.

This is a true story, and it happened last week. I felt so strongly about it that I wrote the entire entry in one sitting.

Please give it a read through, and remember to rate it using the stars! Thanks!

My company has decided that I need to learn more about administration of the Postgres database…which is to say that I should learn something about it. My knowledge is really pretty scant at the moment.

To that end, they’re sending my boss, my junior admin, and me to PostgreSQL East, a conference held in Philadelphia from March 25-28th. We’re doing the conference thing, plus doing training on Sunday.

Anyone out there attending, too?

New blog post up on O'Reilly's Author Blogs.... Tom Limoncelli http://EverythingSysadmin.com

Before I start, I just want you to know that I’m not whining, I just thought I’d give this as an example of some of the things that people who run small infrastructures are left out of…

Today I’m sitting in the office in NJ, doing work as normal. What I’d prefer to be doing is going to the IT Roadmap Conference & Expo in NYC. According to the website, it’s “designed for IT professionals who want to cover multiple industry topics in one day”. That sounds like something I’d be interested in!

Essentially, it’s a sales pitch, or a series of sales pitches. I don’t know if I’m in the market for what they’re selling, but I’d like to go find out what is being offered. All the same, I like to keep my eyes on the horizon, because things have a habit of coming up quick on us in IT, and if we don’t familiarize ourselves with the likely technology of the next few years, then we’ll be caught with our pants down. So I wanted to see what people were selling.

The conference is free. All you have to do is fill out the application for registration. Unfortunately, I don’t qualify:

Dear Matt,

Thank you for your interest in Network World Live’s IT Roadmap Conference & Expo in New York.

Unfortunately, after reviewing the information that you submitted, we determined that at this time, we are not able to confirm your seat on a complimentary basis.

As we noted on the registration form, this event is geared towards network and IT professionals in end-user type companies who actively purchase products and services – or – who will be doing so in the near future. We have a limited number of complimentary seats reserved for attendees who meet this criteria.
…
snip
…
Walk-ins or ineligible applicants arriving at the conference facility will NOT be admitted on the day of the event.
Thank you,

IT Roadmap Team
Network World Events & Executive Forums

(emphasis theirs)

Well, I do actively purchase technologies and products, but not at the scale that they’re looking for, I suppose. I don’t have 50 data centers, or “20,000 or more” servers, so I don’t get to go to their party and look at the toys.

It’s unfortunate for them and me, but somehow I think I’ll live. I just wanted to give you a tangible example of…well…I won’t go so far as to say discrimination, but maybe exclusion, that we small admins deal with from vendors.

If you were waiting to register until the complete schedule was revealed, get that credit card out! LOPSA PICC last night published the final slate of papers and speakers (if you didn't get your accept/sorry email, please let us know).... Tom Limoncelli http://EverythingSysadmin.com

Or as (ir)regular as they normally are. I really hope that you enjoyed the flashback week, and got something useful from it. I’m going to try to do it again next year on the first full week of March.

Now it’s just back to the daily grind for me. I’ve been rehashing some Nagios configuration and I’ve unearthed an ancient relic! How fun! Configuration archaeology is a hobby of mine, and to find a gem that hasn’t (as far as I can tell) been mentioned on the official site since 2002? That’s GREAT! I’ve still got to go through the source code to make sure that it doesn’t do anything interesting, but it’s out of my config now.

As it turns out, my recent attention to Nagios is multifaceted. I’m cleaning up the config and tightening up the alert rules, but also, I’m going to be giving a 45 minute talk at the Professional IT Community Conference in May. If you’re in the northeast US, you should definitely make it! And you should hurry and register while the early bird special is going!

The Threat

Bruce Schneier’s blog post about the Mariposa Botnet has an interesting discussion in the comments about how to make a secure system [1]. Note that the threat is considered to be remote attackers, that means viruses and trojan horses – which includes infected files run from USB devices (IE you aren’t safe just because you aren’t on the Internet). The threat we are considering is not people who can replace hardware in the computer (people who have physical access to it which includes people who have access to where it is located or who are employed to repair it). This is the most common case, the risk involved in stealing a typical PC is far greater than the whatever benefit might be obtained from the data on it – a typical computer user is at risk of theft only for the resale value of a second-hand computer.

So the question is, how do can we most effectively use free software to protect against such threats?

The first restriction is that the hardware in common use is cheap and has little special functionality for security. Systems that have a TPM seem unlikely to provide a useful benefit due to the TPM being designed more for Digital Restrictions Management than for protecting the user – and due to TPM not being widely enough used.

The BIOS and the Bootloader

It seems that the first thing that is needed is a BIOS that is reliable. If an attacker manages to replace the BIOS then it could do exciting things like modifying the code of the kernel at boot time. It seems quite plausible for the real-mode boot loader code to be run in a VM86 session and to then have it’s memory modified before it starts switches to protected mode. Every BIOS update is a potential attack. Coreboot replaces the default PC BIOS, it initialises the basic hardware and then executes an OS kernel or boot loader [2] (the Coreboot Wikipedia page has a good summary). The hardest part of the system startup process is initialising the hardware, Coreboot has that solved for 213 different motherboards.

If engineers were allowed to freely design hardware without interference then probably a significant portion of the computers in the market would have a little switch to disable the write line for the flash BIOS. I heard a rumor that in the days of 286 systems a vendor of a secure OS shipped a scalpel to disable the hardware ability to leave protected mode, cutting a track on the motherboard is probably still an option. Usually once a system is working you don’t want to upgrade the BIOS.

One of the payloads for Coreboot is GRUB. The Grub Feature Requests page has as it’s first entry “Option to check signatures of the bootchain up to the cryptsetup/luksOpen: MBR, grub partition, kernel, initramfs” [3]. Presumably this would allow a GPG signature to be checked so that a kernel and initrd would only be used if they came from a known good source. With this feature we could only boot a known good kernel.

How to run User Space

The next issue is how to run the user-space. There has been no shortage of Linux kernel exploits and I think it’s reasonable to assume that there will continue to be a large number of exploits. Some of the kernel flaws will be known by the bad guys for some time before there are patches, some of them will have patches which don’t get applied as quickly as desired. I think we have to assume that the Linux kernel will be compromised. Therefore the regular user applications can’t be run against a kernel that has direct hardware access.

It seems to me that the best way to go is to have the Linux kernel run in a virtual environment such as Xen or KVM. That means you have a hypervisor (Xen+Linux or Linux+KVM+QEMU) that controls the hardware and creates the environment for the OS image that the user interacts with. The hypervisor could create multiple virtual machines for different levels of data in a similar manner to the NSA NetTop project, not that this is really a required part of solving the general secure Internet terminal problem but as it would be a tiny bit of extra work you might as well do it.

One problem with using a hypervisor is that the video hardware tends to want to use features such as bus-mastering to give best performance. Apparently KVM has IOMMU support so it should be possible to grant a virtual machine enough hardware access to run 3D graphics at full speed without allowing it to break free.

Maintaining the Virtual Machine Image

Google has a good design for the ChromiumOS in terms of security [4]. They are using CGroups [5] to control access to device nodes in jails, RAM, CPU time, and other resources. They also have some intrusion detection which can prompt a user to perform a hardware reset. Some of the features would need to be implemented in a different manner for a full desktop system but most of the Google design features would work well.

For an OS running in a virtual machine when an intrusion is detected it would be best to have the hypervisor receive a message by some defined interface (maybe a line of text printed on the “console”) and then terminate and restart the virtual machine. Dumping the entire address space of the virtual machine would be a good idea too, with typical RAM sizes at around 4G for laptops and desktops and typical storage sizes at around 200G for laptops and 2T for new desktops it should be easy to store a few dumps in case they are needed.

The amount of data received by a typical ADSL link is not that great. Apart from the occasional big thing (like downloading a movie or listening to Internet radio for a long time) most data transfers are from casual web browsing which doesn’t involve that much data. A hypervisor could potentially store the last few gigabytes of data that were received which would then permit forensic analysis if the virtual machine was believed to be compromised. With cheap SATA disks in excess of 1TB it would be conceivable to store the last few years of data transfer (with downloaded movies excluded) – but such long-term storage would probably involve risks that would outweigh the rewards, probably storing no more than 24 hours of data would be best.

Finally in terms of applying updates and installing new software the only way to do this would be via the hypervisor as you don’t want any part of the virtual machine to be able to write to it’s data files or programs. So if the user selects to install a new application then the request “please install application X” would have to be passed to the hypervisor. After the application is installed a reboot of the virtual machine would be needed to apply the change. This is a common experience for mobile phones (where you even have to reboot if the telco changes some of their network settings) and it’s something that MS-Windows users have become used to – but it would get a negative reaction from the more skilled Linux users.

Would this be Accepted?

The question is, if we built this would people want to use it? The NetTop functionality of having two OSs interchangeable on the one desktop would attract some people. But most users don’t desire greater security and would find some reason to avoid this. They would claim that it lowered the performance (even for aspects of performance where benchmarks revealed no difference) and claim that they don’t need it.

At this time it seems that computer security isn’t regarded as a big enough problem for users. It seems that the same people who will avoid catching a train because one mugging made it to the TV news will happily keep using insecure computers in spite of the huge number of cases of fraud that are reported all the time.

• [1] http://www.schneier.com/blog/archives/2010/03/mariposa_botnet.html
• [2] http://www.coreboot.org/Welcome_to_coreboot
• [3] http://grub.enbug.org/FeatureRequests
• [4] http://sites.google.com/a/chromium.org/dev/chromium-os/chromiumos-design-docs/system-hardening
• [5] http://lxr.linux.no/linux+v2.6.30/Documentation/cgroups/

In a comment on my post Shared Objects and Big Applications about memlockd [1] mic said that they use memlockd to lock the entire root filesystem in RAM. Here is a table showing my history of desktop computers with the amounts of RAM, disk capacity, and CPU power available. All systems better than a 386-33 are laptops – a laptop has been my primary desktop system for the last 12 years. The columns for the maximum RAM and disk are the amounts that I could reasonably afford if I used a desktop PC instead of a laptop and used the best available technology of the day – I’m basing disk capacity on having four hard drives (the maximum that can be installed in a typical PC without extra power cables and drive controller cards) and running RAID-5. For the machines before 2000 I base the maximum disk capacity on not using RAID as Linux software RAID used to not be that good (lack of online rebuild for starters) and hardware RAID options have always been too expensive or too lame for my use.

Year CPU RAM Disk Maximum RAM Maximum Disk 1988 286-12 4M 70M 4M 70M 1993 386-33 16M 200M 16M 200M 1998 Pentium-M 233 96M 3G 128M 6G 1999 Pentium-2 400 256M 6G 512M 40G 2000 Pentium-2 600 384M 10G 512M 150G 2003 Pentium-M 1700 768M 60G 2048M 400G 2009 Pentium-M 1700 1536M 100G 8192M 4500G 2010 Core 2 Duo T7500 2200 5120M 100G 8192M 6000G

The above graph shows how the modern RAM capacities have overtaken older disk capacities. So it seems that a viable option on modern systems is to load everything that you need to run into RAM. Locking it there will save spinning up the hard drive on a laptop. With a modern laptop it should be possible to lock most of the hard drive contents that are regularly used (IE the applications) into RAM and run with /home on a SD flash storage device. Then the hard drive would only need to be used if something uncommon was accessed or if something large (like a movie) was needed. It also shows that there is potential to run diskless workstations that copy the entire contents of their root filesystem when they boot so that they can run independently of the server and only access the server for /home.

Note that the size of the RAM doesn’t need to be larger than the disk capacity of older machines (some of the disk was used for swap, /home, etc). But when it is larger it makes it clear that the disk doesn’t need to be accessed for routine storage needs.

I generated the graph with GnuPlot [2], the configuration files I used are in the directory that contains the images and the command used was “gnuplot command.txt“. I find the GnuPlot documentation to be difficult to use so I hope that this example will be useful for other people who need to produce basic graphs – I’m not using 1% of the GnuPlot functionality.

• [1] http://etbe.coker.com.au/2009/12/19/shared-objects-and-big-applications/
• [2] http://www.gnuplot.info/

Tom's presentation is TBD. (Including this with the "appearances" tag so it shows up on the navigation)... Tom Limoncelli http://EverythingSysadmin.com

Tom will be the Saturday opening keynote, plus he will be teaching his two most popular half-day classes: Time Management for System Administrators, and "Help! Everyone hates our IT department!". LOPSA NJ PICC is in New Brunswick, NJ, May 7-8,... Tom Limoncelli http://EverythingSysadmin.com

Click the cartoon for more information!... Tom Limoncelli http://EverythingSysadmin.com

You are probably a human. At least, the statistical odds are in your favor. As a human, you experience stress, and how you react to it plays a large part in determining how happy you are. System administrators deal with stress particularly poorly, in general. We assume the role of hero and that’s that. Do what it takes, bask in whatever glory accompanies the successful completion of our task.

There is no downtime in that equation. Immediately following those emergencies, most of us drink depressants to bring ourselves down. On normal days, we require morning stimulants to bring ourselves up. I highly suspect that some of us are so called “adrenaline junkies” from the relative high that we get when there’s an immediate problem that no one can solve but ourselves.

This is unhealthy.

What we really need is to be able to step back and look at the pattern in our lives and say I don’t want to live with this stress.

When it first hit me that stress is probably the biggest single microproblem for admins, I wrote the following. I hope you find it relevant.

Jack Hughes, over at the Tech Teapot, mentions a very appropriate subject for too many systems administrators: burnout.

As sysadmins, we’re nearly always the go-to person for whatever happens. After a while, we start to get used to it, and lots of times, we can develop a hero complex, carrying the weight of the world on our shoulders, at least in our minds. This isn’t healthy for a lot of reasons, the most important of which is your health.

Here’s an example of what taking your job too seriously can do to you:

Part One

Part Two

Not to ruin the ending, but the most disgusting part is that, while the guy was taking medical leave, his company fired him. To be completely honest, he’s much better off without a company like that, and if your company would do the same thing, then so are you.

To quote Peter Gibbons, “We don’t have a lot of time on this earth. We weren’t meant to spend it this way. Human beings were not meant to sit in little cubicles staring at computer screens all day…”

Even one of the most preeminent Systems Administrators around, Tom Limoncelli advocates leaving the pressure at work when you head home. For those of us on call 24/7/365, that can be a little hard, but it’s important to try.

Tonight's topic is "What's the biggest problem in system administration?" his month's meeting will be less technical, more philosophical.What's the biggest problem facing system administrators? Is it the vendors? The managers? The tools? Is it us? (nah, it couldn't be us!... Tom Limoncelli http://EverythingSysadmin.com

The Opera-Mini Dispute

I have just read an interesting article about the Opera browser [1]. The article is very critical of Opera-Mini on the iPhone for many reasons – most of which don’t interest me greatly. There are lots of technical trade-offs that you can make when designing an application for a constrained environment (EG a phone with low resolution and low bandwidth).

What does interest me is the criticism of the Opera Mini browser for proxying all Internet access (including HTTPS) through their own servers, this has been getting some traction around the Internet. Now it is obvious that if you have one server sitting on the net that proxies connections to lots of banks then there will be potential for abuse. What apparently isn’t obvious to as many people is the fact that you have to trust the application.

Causes of Software Security Problems

When people think about computer security they usually think about worms and viruses that exploit existing bugs in software and about Trojan horse software that the user has to be tricked into running. These are both significant problems.

But another problem is that of malicious software releases. I think that this is significantly different from Trojan horses because instead of having an application which was written for the sole purpose of tricking people (as is most similar to Greek history) you have an application that was written by many people who genuinely want to make a good product but you have a single person or small group that hijacks it.

Rumor has it that rates well in excess of $10,000 are sometimes paid for previously unknown security vulnerabilities in widely used software. It seems likely that a programmer who was in a desperate financial situation could bolster their salary by deliberately putting bugs in software and then selling the exploits, this would not be a trivial task (making such bugs appear to be genuine mistakes would take some skill) – but there are lots of people who could do it and plausibly deny any accusation other than carelessness. There have been many examples of gambling addicts who have done more foolish things to fund their habit.

I don’t think it’s plausible to believe that every security flaw which has been discovered in widely used software was there purely as the result of a mistake. Given the huge number of programmers who have the skill needed to deliberately introduce a security flaw into the source of a program and conceal it from their colleagues I think it’s quite likely that someone has done so and attempted to profit from it.

Note that even if it could be proven that it was impossible to profit from creating a security flaw in a program that would not be sufficient to prove that it never happened. There is plenty of evidence of people committing crimes in the mistaken belief that it would be profitable for them.

Should We Trust a Proprietary Application or an Internet Server?

I agree with the people who don’t like the Opera proxy idea, I would rather run a web browser on my phone that directly accesses the Internet. But I don’t think that the web browser that is built in to my current smart-phone is particularly secure. It seems usual for a PC to need a security update for the base OS or the web browser at least once a year while mobile phones have a standard service life of two years without any updates. I suspect that there is a lot of flawed code running on smart phones that never get updated.

It seems to me that the risks with Opera are the single point of failure of the proxy server in addition to the issues of code quality while the risks with the browser that is on my smart-phone is just the quality of the code. I suspect that Opera may do a better job of updating their software to fix security issues so this may mitigate the risk from using their proxy.

At the moment China is producing a significant portion of the world’s smart-phones. Some brands like LG are designed and manufactured in China, others are manufactured in China for marketing/engineering companies based in Europe and the US. A casual browse of information regarding Falun Gong makes the character of the Chinese leadership quite clear [2], I think that everything that comes out of China should be considered to be less trustworthy than equivalent products from Europe and the US. So I think that anyone who owns a Chinese mobile phone and rails against the Opera Mini hasn’t considered the issue enough.

I don’t think it’s possible to prove that an Opera Mini with it’s proxy is more or less of a risk than a Chinese smart-phone. I’m quite happy with my LG Viewty [3] – but I wouldn’t use it for Internet banking or checking my main email account.

Also we have to keep in mind that mobile phones are really owned by telephone companies. You might pay for your phone or even get it “unlocked” so you can run it on a different network, but you won’t get the custom menus of your telco removed. Most phones are designed to meet the needs of telcos not users and I doubt that secure Internet banking is a priority for a telco.

Update: You can buy unlocked mobile phones. But AFAIK the Android is the only phone which might be described as not being designed for the needs of the telcos over the needs of the users. So while you can get a phone without custom menus for a telco, you probably can’t get a phone that was specifically designed for what you want to do.

The Scope of the Problem

Mobile phones are not the extent of the problem, I think that anyone who buys a PC from a Chinese manufacturer and doesn’t immediately wipe the hard drive and do a fresh OS install is taking an unreasonable risk. The same thing goes for anyone who buys a PC from a store where it’s handled by low wage employees, I can imagine someone on a minimum income accepting a cash payment to run some special software on every PC before it goes out the door – that wouldn’t be any more difficult or risky than the employees who copy customer credit card numbers (a reasonably common crime).

It’s also quite conceivable that any major commercial software company could have a rogue employee who is deliberately introducing bugs into it’s software. That includes Apple. If the iPhone OS was compromised before it shipped then the issue of browser security wouldn’t matter much.

I agree that having the minimum possible number of potential security weak points is a good idea. They should allow Opera Mini users to select that HTTPS traffic should not be proxied. But I don’t think that merely not using a proxy would create a safe platform for Internet banking. In terms of mobile phones most things are done in the wrong way to try and get more money out of the users. Choose whichever phone or browser you want and it will probably still be a huge security risk.

Harald Welte is doing some really good work on developing free software for running a GSM network [4]. But until that project gets to the stage of being widely usable I think that we just have to accept a certain level of security risk when using mobile phones.

• [1] http://counternotions.com/2010/02/18/mini/
• [2] http://www.faluninfo.net/
• [3] http://etbe.coker.com.au/2009/05/31/lg-u990-viewty/
• [4] http://laforge.gnumonks.org/weblog/2010/02/13/#20100213-six_weeks_to_bcch

It’s the end of a long day. You lean back in your chair, sigh, and you’re glad it’s time to go home. Someone asks you what you did all day. You just sort of shake your head and say “fought fires”.

Fire fighting, as a sysadmin, means you don’t make any progress. You only work very hard to stay where you are. Working against entropy is difficult, and it can take a lot out of you. Some days are harder than others.

One day in early June, not long after I started this blog, I experienced a major setback. Also, a major power outage. Our entire backup facility lost power, and what’s worse, the generator refused to kick on. Our secondary site was down hard for days, until the power was restored to the downtown area of the village we were located in.

During the problem, though, we were able to turn a major issue into a net gain. Read on for the rest of the story…

It’s funny, sometimes, how we tolerate suboptimal or downright malproductive arrangements in our infrastructures, just because it’s inconvenient or inopportune to do it the “right way”. It seems like “the right way” either never comes, due to projects getting phased out, or it gets fixed during a cataclysmic upheaval, when it has become an immediate concern.

The case in point is my mail server. We have an A and a B mx record. Originally the B MX just stored mail until the A came back up, then it would get delivered. Everyone checks mail on A, so it can’t really be down during the day, and about 6 months ago, the office that B was at relocated and B was never set up. This left us with just A. To make matters worse, A was old enough that it was physically located in our backup site, which used to be our primary site. This was suboptimal. Of course there was talk about moving it to the primary site, but when could a maintenance window be created? And we’d risk the entire period of non-connectivity when it was being moved. No, management said, lets just leave it where it was.

Great strategy. It actually worked fine though, until this weekend.

I came in on Saturday, ready to do some major work on the blade systems I’m building for our new site. I sat down at my desk, ready to dive into work. Since I was alone, Raiders of the Lost Ark was playing on the laptop. I had just logged into the first server when the lights went off, and the telltale screech and whine from the server room told me that we’d lost main power.

In Granville, OH, that’s not a strange thing. We’ve got backup AC and a backup generator, so I wasn’t worried. It does have to be manually started, so I jogged into the server room and turned on the CFL floor lamp. At least I tried to. I looked at the generator control panel and it confirmed my fears. No generator power.

I tried for several minutes to start it, but nothing gave me the impression that anything would change, so I called my boss to let him know the situation, and that I was going to start shutting down machines. Since the only critical thing was mail, I suggested that he change DNS to point to an as-yet unassigned IP at the colocation, and that I could setup a postfix process there to queue the mail. He said that it would work, but he suggested an alternative approach.

Why not relocate the physical mail server to the colocation? A lightbulb went off. Of course, not only could I take care of that long standing problem, but because there was no power at all in the datacenter, the normal policy of no-downtime-for-repairs-and-upgrades was out the window.

The next morning, I left work to go home at 5am. The previous 15 hours had been spent completely rehauling the backup datacenter. With the mail relocated to the primary facility, once the power came on in the backup, I had free reign to cull everything unnecessary that had been accumulating.

There is now a pile of cables covering a square yard or so around 6 inches deep of power, ethernet, and copper/fiber cables. There are something like 96 ports worth of switches that I took out, multiple servers, KVMs, fiber switches, and general cruft. The servers are also arranged so that no half-depth servers are hiding between full depth. That was always a pet peeve of mine.

I thought about it while I was doing this, and if fighting normal issues is considered firefighting, then what I went through should have been considered forestfire fighting. And just like a forest fire, good can come from it. It takes the massive heat of a forestfire to crack open some pine cones. It also takes massive infrastructure downtime to make significant changes.

A series of pictures taken last month in San Francisco.... Tom Limoncelli http://EverythingSysadmin.com

This is a short bit that I wrote when I was considering overhauling the internal naming scheme at my company. We used to use an odd mismash of names, and we used to have multiple invented internal DNS names, that referred to the physical location. And I don’t mean things like “location.example.com” (that might make sense!). I mean it would be as if General Motors had “boston.gm” and “tijuana.gm” and “tokyo.gm”. Nonesensical in a lot of ways (particularly now that the TLD’s can be bought for a song (well, an expensive song)).

Anyway, I was curious how other people did it, so I asked. As it turns out, this post originally aired in July of 2008. I would guess that I had a couple of hundred readers. That’s a good range of experience to draw from, but I wanted a more broad view, so I submitted it to slashdot. And it got on the front page.

Thanks to Slashdot, this entry originally received 43 comments, which is right around 30 more than the next most popular story at that point. I’ve had a lot of people tell me that they found me because of that front page article. I didn’t submit it to drive people to the blog; I really did want to hear what people were doing with their own networks. Driving people to the blog was a completely satisfactory side effect, though

Before you leave this page, make sure to check out the original and read the comments. There’s a lot of funny (and interesting) ideas!

Enjoy!

Bob Plankers, over at The Lone Sysadmin wrote a couple days ago about getting busted while reading the wiki page on X-Men. He tried to cover it up by claiming to be researching future host names. Quick thinking, Bob. Good job!

It does bring up a good point, though. Internal naming schemes are something that everyone has an opinion on, and a load of suggestions.

At various places, I’ve used greek/roman gods, Simpsons characters, beer companies, wine labels, and fish.

At my current company, we used the beer and wine names. We absorbed another company that used fish. It worked fine for a while, but we grew in terms of servers and locations until it got unwieldy to remember A) all the names, and B) what each name did. You’d also start to get very similar names after a while. We’ve now got 4 physical locations, soon to be 5, and something like 50-60 servers (not counting network devices), no one would be able to keep them all straight (including the admin).

To improve the situation, we’re in the process of changing to location-based hostnames with a flat internal domain structure. For example, the 2ndary application server in Ohio is oh-app2, with the fake internal domain name trailing. The alpha site’s primary fileserver is a-fs1.

It’s no where near as fun as “wolverine.internal.com” but it certainly does tell you where you’re connecting to and what the machine does. What makes it interesting is when you go changing things like CVS repositories on people’s machines, mail servers, etc. The policy we’ve taken is to alias the old information to the new, and slowly phase out the old method.

What do you use as internal naming systems? What do you think would make an excellent scheme? Make sure to check the list to make sure it hasn’t been done before!