Like many other sysadmins, I'd rather have Solaris or Redhat/Centos as my server OS of choice than Ubuntu. Debian is great too, but I'm used to the Redhat way of configuration.
post a comment
I was once a big fan of Ubuntu. Then I discovered Gentoo, and ended up learning a bit about setting up a desktop environment from scratch.
Despite rumours of a fall in package quality and of a decline in testing, people continue to use Ubuntu. I thought of jotting down some points here on why I feel this may be so:
1. bash command not found handler
2. Good desktop integration
This is almost as good as what OSX or Windows 7 have today by way of MIME type handling, file associations, etc.
3. Good collection of updated packages
The wide variety of packages help users understand and have confidence that they can do a wide variety of things with Ubuntu.
Other distributions (Linux, BSD-style) too have a variety of packages, but the public perception is that the Ubuntu package repositories are very much up to date, and that they have a wide variety of packages.
There are some other points that set Ubuntu apart. See
Some of these points can be fixed on Belenix and on other distros using technology, while others need a lot of perception management.
Update: Removed the Music label. The opensolaris planet was picking up the music label as the title of the post !
Given the uncertainity around the OpenSolaris 2010.03 release and the absence of any clear information from Oracle on the opensolaris code base, the OpenSolaris distro, and the opensolaris community, it is but natural to wonder as to what would happen next.
1 comment | post a comment
Moinak and I have been discussing these topics ourselves nearly everyday.
Our thoughts are as follows:
- Let the OGB meet together, and perhaps also meet a senior Oracle person such as Dan Roberts, and understand what Oracle's plans are as well as the reason for the silence.
The OGB represents the community, and is supposed to represent and oversee the Community Groups, so it's best the OGB take optimistic and friendly steps toward Oracle.
- It is not yet time to fork the code base. Let us learn conclusively that Oracle will not be contributing code, and then take a decision along with the opensolaris community.
Our plans at present are to get the Belenix 0.8 release out, and to then work with the community and the OGB to talk to Oracle and decide on further steps.
I'm looking forward to giving my talk at the Sun Tech Days the coming week.
2 comments | post a comment
Rolling Solaris into the organization has been an interesting adventure. Buying a Fishworks unit was not working out, though I tried to make that happen twice. So, onto Plan B.
- Set up a cheap storage box using Solaris 10.
- Export NFS mount points and backup build artifacts onto that.
- Export iSCSI LUNs and use for VMWare
- Coach fellow sysadmins every step of the way, and make them manage these boxes
- Show how fast setting up VM eco systems can be when you use ZFS snapshots and clones (e.g. six VM environments ready for configuration in < 5 mins).
- Move on to Zones :)
Confidence building was the biggest task. I was helped by a few disks failing, and my colleagues discovering how awesome ZFS repairs can be.
We also use Linux DM based mirroring (since LVM mirroring is not upto the mark). ZFS mirroring speeds are easy to convince anyone, when they see even creating an empty 1 TB ext3 DM mirror can take upto six hours.
A very important thing that I've learned is that evangelizing Solaris technologies to the average Indian sysadmin requires a lot of Indianization, where we speak of Indian realities (dealing with people, management, budgets).
I'm looking forward to my talk on all of this at the Sun Tech Days on Thursday.
This is the last Sun Tech Days in India. Later, perhaps, it'll be part of the Javaone in India, perhaps as part of the "Develop" event.
Here's a snapshot of my desktop which I use at work
post a comment
That's Belenix 0.8 Alpha running KDE 4.2.4 compiled with GCC 4.4. I backed up my data, installed OpenSolaris 2009.06, and then ran the install_belenix script after modifying it to point to my local Belenix mirror. Install took 20 minutes, and I now have OpenSolaris as well as Belenix 0.8 Alpha in my GRUB boot menu.
I've taken Wine from Triskelios' builds since we need to patch GCC 4.4.1 in order to get it to compile Wine on Belenix. This is good enough to run Lotus Notes 7.0.1. I do face two errors at startup related to z:\, which I've not got around fo fixing yet.
Also, I've taken rdekstop from www.sunfreeware.com since I needed to get rdesktop running first and didn't have time to set up a GCC 4.4 based dev environment (which I now have).
Some bugs that I've noticed so far:
- The KDE Lock tool doesn't let me unlock ! This is a critical bug which need some investigation.
- The cursor in Konsole is offset by a fre characters from the point of actual character entry.
- KWin sometimes thinks that the Alt key is continuously pressed. I "solve" this by either restarting KDE (Control Alt Backspace), or I SSH into my box and kill the startkde4 process. I've had to do this four times during the past week.
We have core dumps for the above and are considering either investigating further, or simply moving to KDE 4.3
Other bugs that I've noticed include:
- Okular doesn't support PDFs due to some issues with how poppler has been built.
- On a colleague's laptop, the CPU fan started to run at full speed. This is the first time that I've notived this issue.
There may be other issues too, but I've not made time to test them yet.
We've ofcourse had a number of bugs with the install_belenix script, each of which has been fixed.
We welcome you to try out Belenix 0.8 Alpha and report issues. Even if you have OpenSolaris already installed, you can go ahead and install Belenix since it'll neither damage your existing data, nor format anything. The script will ask you for a user name, and will create a separate user account for you.
chmod +x install_belenix
post a comment
What many may not know, is that having great disk i/o and a good file system can help in certain situations.
So also with an excellent network stack.
As a sysadmin who doesn't use Windows at all, I have to RDP over to a Windows server, start IE on that computer, and administer switches from there.
2 comments | post a comment
There are also sites like www.cisco.com which are not Firefox friendly at all.
I recently wrote to the marketing team at Promise (the SATA Controller company), pointing out that their website is not usable by non-IE users. They wrote back stating that the website will be revamped soon.
At my company, the marketing team tests all functionality to ensure that mose browsers that we can think of work fine. This includes IE 6/7/8, Safari (OSX/Windows), Firefox (OSX/Windows/Belenix). We don't test with Konqueror or with Webkit yet, though I think those two should be included too.
As a developer, I know that web standards compliance requires a bit of diligence but that this can be achieved.
On Friday night at work, some of us were discussing our plans for the weekend. I mentioned that I'd visit I2IT - a college in Pune - and give a presentation on OpenSolaris and ZFS to the students. Ever on the watch for a chance to showcase ZFS, I gave my colleagues a quick demo on ZFS.
2 comments | post a comment
We had a look at creating a snapshot, deleting files, and then recovering files from the .zfs folder as well as by rolling back to the snapshot. I also cited some disk performance numbers (5 mins for a full SVN checkout of a particular project on my laptop, vs 30+ mins for others who use Windows on the same laptop model). I also mentioned the notion of pools, of how one can transparently add storage, etc.
I will be showcasing ZFS sometime this week to the entire office.
While we were leaving, my colleague remarked "ZFS is the most amazing technology I've seen in recent times".
Reflecting on it a bit, I couldn't agree more !
Here is a useful link that helps explain Cloud Computing : http://industry.bnet.com/technology/10001746/fog-is-lifting-on-cloud-computing/
post a comment
The Open Cloud Consortium has as list of software related to Cloud Computing. There may be others too.
According to the statements on Sector listed at that page, Sector is supposed to be twice as fast as Hadoop .
Though I consider myself too old to understand the medium, I'm intrigued by the effort needed to keep Facebook running. Facebook's cloud computing related software, called Thrift , is available for download too.
Matt Asay asks Cloud computing: A natural conclusion of opensource ? . This is an informative read, because Matt provides perspectives on how users are no longer interested in the underlying technology, but in how they access that technology, and how inter-operable the data is.
My roommate is excited about opensolaris. Being a .NET developer, he wished to help create a mono package for Belenix. While I walked him through an exercise on building mono from source, I realized that we had a "broadband" line which gives us just 20KBps. While trying to get mono to build (we've not done this yet due to a corlib.dll version mismatch with mcs), we had to wait a while for mono and for monolite to get downloaded.
6 comments | post a comment
Yesterday, I'd presented on Belenix and ZFS at a local college I2IT, and had fallen short on DTrace and Zones demos. My custom Belenix installation has a broken zones config, and I've not made time to learn to write DTrace scripts.
I also learned that it is impossible for the students of I2IT to download the Belenix ISO, or any of the tools and software that one could run on. Though they have a Sun Campus Ambassador who is doing a good job, he can only distribute what he receives as part of the ambassador program. Anything more, and the students have to fend for themselves.
All opensolaris evangelists should have a good collection of demos for various topics, familiarity with the various topics that they are going to demo, CDs/DVDs to give away, and learning material (or at least a printout of URLs that people should visit - in case you run out of CDs/DVDs).
Expecting people to download content is unrealistic - they may not have the bandwidth, and the sheer chore of having to download something could be a deterrent. This is a reality in India, and when we evangelize, we need to have various tools ready and available on distribution media.
It may be worth having a demo CD all by itself, in case the basic Belenix CD does not have space. An alternative to consider would be having DVDs.
Here are some demos that I think we should have, along with the requisite software where possible.
- Entertainment, Games, Graphics tools, OpenOffice
- Communication Tools (IRC, Browsers, streaming-content servers and clients, file download tools, Skype in a Linux Zone, Jabber, etc). See note below on Skype.
- Web app stacks
- Wine for running Windows apps, along with some free Windows apps
- Development Tools
- Using opensolaris tools to solve college assignments, and to understand subjects better.
We could have sample code that illustrates a topic, and DTrace scripts that helps the programmer see how the code behaves on a running system. This would be a very useful demo, imo.
A lot of the above is already available thanks to the efforts of the opensolaris community, the various blogs, and the documentation team. The next round of work would be to collate all this together, improve where necessary, and to create DVDs. There'd be a lot of testing too.
The creators/communities of the various tools would be a good group to source demos from.
Along with all this, we should also have instructions on setting up a local mirror, a DVD of the Belenix repository which could be used as the seed for the local mirror.
Note on Skype redistribution: We're allowed to re-distribute Skype, provided we send them an email, and comply with various conditions there ( a bit of a task, but can be achieved).
Would it be legal to setup Skype in a zone and distribute that ? We should find out.
Note on Sun Studio Redistribution:
We need to have an answer once and for all - Can the excellent Sun Studio Compilers be redistributed by non-Sun distros ? We've initiated conversations by have not followed up on them.
Amey and I presented on opensolaris at Pune yesterday. Amey has put up a report here
post a comment
Here's my own analysic.
What went well:
- Rapport with the audience
- ZFS demos
In my experience with showcasing opensolaris, ZFS alone sometimes convinces people that opensolaris should be considered seriously.
To Sun's ZFS team: I hope to do you proud with even better ZFS demos !
What didn't go well:
- Some students "forgot" that I'd mentioned that Belenix was a first class KDE 3.5.x environment.
They wanted to know at the end how Belenix compared to a KDE environment. I blame myself for such questions, since I should have established beyond doubt that Belenix has just about everything that a KDE environment has to offer.
- I forgot to prepare and take along Belenix CDs :(
I was simply flooded with work, and am anyway guilty of not managing my time well.
Now that I'm used to a 10 Mbps line at work where ISOs are downloaded within an hour or so, and because I know that I2IT is a premier institution, I assumed that the students would be able to download the Belenix ISO themselves. They told me that this was not possible given that they had limited bandwidth at college and that they had to go through a committee for anything over 30 MB in size. This excludes even GParted, btw, which is over 30 MB in size.
This made me realize that if I get used to having high-speed bandwith despite being an Indian staying in India, it is unrealistic to expect the IPS team to understand that the world cannot download lots of content all the time.
- Emphasize how Belenix is a first class KDE environment.
Talk to the KDE evangelizing team for ideas here.
Mention how the next version of Belenix could set the gound for opensolaris being _the_ KDE platform of choice.
- Different presentation styles for students vs working professionals.
- Demos, demos, demos
This needs a separate blog post by itself.
- Have CDs and the opensolaris Learning Guides ready for use.
Bottom line: Be prepared.
One blog post that I read just now is "Complexities of APIs" where Ketan observes how a service locator gets in his way.
post a comment
One of the many things I enjoy taking care of as a sysadmin are the SVN repositories at our Bangalore office. These are large repositories, and have three known mirrors, one NFS share where each commit is incrementally dumped, as well as a secret mirror that no one else knows about ;)
post a comment
Here are the steps that I follow when setting up an SVN mirror. I'll later post information on problems that one faces when setting up and maintaining SVN mirrors.
Very, Very, Important: When committing to an SVN mirror, you need to use TortoiseSVN 1.5.0 and no later. Any later revision may work, but may break during certain tasks. There is no such problem when committing to the main SVN server.
How SVN mirroring works (in a nutshell):
Users connect to mirror servers. mod_dav_svn serves content from the local system, and sends commits to the main server. The main server's post-commit scripts use svnsync to push commits to the mirrors via a protected URL that is writable only from the main server.
My instructions include integrating with LDAP on an Active Directory server.
Install CentOS 5.2 (I've tried FC8 and Ubuntu, and they have some wierd issue or the other with SVN mirroring. Nothing that I can definitely point out).
Uninstall all SVN the comes with the OS
Install Collabnet SVN client binaries
Install Collabnet SVN server binaries
symlink /opt/CollabNet_Subversion/bin/ into /usr/bin
Include the following httpd fragment on the main server's httpd.conf file. You could Include another file containing the following, too. This is what I do.
LoadModule dav_svn_module /etc/httpd/modules/mod_dav_svn.so
LoadModule authz_svn_module /etc/httpd/modules/mod_authz_svn.so
AuthName "Active Directory LDAP Authentication"
AuthLDAPURL "ldap://adserver.thoughtworks.com:389/ou=Principal,dc=Corp orate,dc=thoughtworks,dc=com?sAMAccountName?sub?(&(objectClass=user))"
Create a repository as follows:
svnadmin create /repos/svn/repos/someproject
change permissions as follows
chmod -R g+w /repos/svn/repos/someproject
chown -R apache.apache /repos/svn/repos/someproject
symlink the collablnet svn modules
ln -s /opt/CollabNet_Subversion/modules/mod_dav_svn.so /etc/httpd/modules/mod_dav_svn.so
ln -s /opt/CollabNet_Subversion/modules/mod_authz_svn.so authz_svn_module /etc/httpd/modules/mod_authz_svn.so
service httpd reload
On the mirror server:
Follow instructions as above.
Modify the httpd Location to become
Also include the following snippet:
Deny from all
Allow from svnserver.thoughtworks.com
cat > /repos/svn/repos/someproject/hooks/pre-revprop-change << EOF
chmod +x /repos/s
Reload the httpd server
service httpd reload
Now, back on the main svn server
- initialize the SVN mirror on the mirror server
svnsync init http://svnmirror.thoughtworks.com/someproject-sync http://svnserver.thoughtworks.com/someproject
Edit the post-commit hook to contain the following:
svnsync sync http://svnmirror.thoughtworks.com/someproject-sync --source-username read_only_user --source-password valid_password
You can now either let the next post-commit update the mirror at one shot, or transport the data yourself and then update the mirror's metadata.
To populate the mirror automatically, simply commit to the main server once.
Populating the SVN mirror manually to seed it with data is the preferred approach, especially when the mirror is at a remote location.
Note down the properties of the mirror server
svn pl --revprop -r 0 http://svnmirror.thoughtworks.com/someproject-sync
Unversioned properties on revision 0:
Get each property:
svn pg svn:sync-from-uuid --revprop -r 0 http://svnmirror.thoughtworks.com/someproject-sync
svn pg svn:sync-last-merged-rev --revprop -r 0 http://svnmirror.thoughtworks.com/someproject-sync
svn pg svn:date --revprop -r 0 http://svnmirror.thoughtworks.com/someproject-sync
svn pg svn:sync-from-url --revprop -r 0 http://svnmirror.thoughtworks.com/someproject-sync
1. Export the svn data
Assuming the latest revision is revision 32478 (an excellent case for populating data manually)
svnadmin dump /repos/svn/repos/someproject -r 1:32478 --incremental | gzip -9 > someproject_1_32748.gz
2. Transfer the file to the mirror server
scp someproject_1_32748.gz firstname.lastname@example.org:
On the remote server:
3. Extract the archive file
4. Import the data into the repository
svnadmin load /repos/svn/repos/someproject < someproject_1_32748
Back on the main server
5. update the mirror repository's sync-last-merged-rev property
svn ps svn:sync-last-merged-rev 32748 --revprop -r 0 http://svnmirror.thoughtworks.com/someproject-sync
6. Ensure that everything's fine
svnsync sync http://svnmirror.thoughtworks.com/someproject-sync
If you face any problems, reply to this post and I'll see how I can help with my own experiences on solving those problems :)
Once I get SVN 1.5 based mirroring to work on Solaris 10, I'll post about that too. In my opinion, Solaris 10 + ZFS should be the best bet for SVN servers and mirrors, because there are some problems that even having a mirror will not help you solve quickly.
Some such problems are:
- what if the ext3 file system goes corrupt (I just dealt with a corrupted ext3 file system on an RHEL 5 box this weekend).
- what if someone makes a dangerous commit and you need to remove that commit before anyone else checks it out ?
one example would be checking in a massive delete/folder structure by mistake
another example would be checking in an infected file
yet another example would be checking in installers into the repository.
If you need to roll back quickly, you can do so if you have ZFS snapshots per post-commit, but not with other filesystems. With other file systems, you'd need to go through the painful steps of dumping all the commits, importing in data upto (but not including) the poison commit, and then pointing users to that repository.
A alternative would be to have another repository which is updated once every twenty commits or so. You would then have the opportunity to selectively push in commits and to then drop the most recent bad commit.
The best would of course be to use Mercurial and to ditch SVN altogether. This is not always feasible, though.
I'm told that there aren't adequate tooling to support Mercurial in Windows as much as there are for SVN. Next, our customers may be averse to moving to (and asking their other vendors to move to) a new tool (we should ask, though).
One lesson that I have realized the value of is "Try to learn from others' mistakes".
post a comment
One such mistake that I hope to not make myself is that of not delivering when I have the best at my disposal - A great management, hard working colleagues, a positive environment, a patient boss, and the opportunity to take us to the next level.
I really hope that I never have to write a mail like Jonathan did, when the entire Sun board had amazing engineers in the world, but could not make it's marketing team deliver - A marketing team which comprises of people who will waste your time criticizing the Linux kernel instead of focussing on Solaris' advantages. (The two persons I've communicated with in India were different, though).
I read a discussion thread recently where an ill-informed person commented that Sun has dumped code on the world and that it's not really an open source leader. We who know and love Sun know better. I didn't argue with him, because here too, one needs to blame the absence of marketing and brand imaging building.
It takes courage to call a spade a spade, and Ben did a good job of this.
A large number of OSX users will be able to demonstrate to you that OSX improves their productivity a lot. Though I don't know a great many of my customers whose productivity has actually improved, there are a limited number of OSX power users I know, and when I see them at work, I am convinced that OSX is good for them.
6 comments | post a comment
However, what most people do not know is the horrible hell that sysadmins have to go through when it comes to supporting the Apple hardware platform.
Our problems are with the warranty, the absence of any insurance policy, and the high price of Apple hardware.
Unlike with Dell hardware, you don't get comprehensive warranty with Apple hardware. You need to prove that the defect that you are facing on Apple hardware is a manufacturing defect, else you need to pay money from your pocket. If you were to accidentally drop your laptop and the case were to get cracked, or if the security fellows at the airport mis-handled your lugguage and the display were to crack, your further experiences would be different depending upon whether you had a Dell laptop or a Mac. With a Dell, you can get a same or next business day replacement if you paid for such a plan (At Thoughtworks, we pay for such a plan).
With a Mac, however, there is no such plan to pay for.
At least in India, no Insurance company is willing to insure Mac hardware, though they have no problems with Dell hardware.
For the price of Apple's hardware, you can get far, far more if you were to opt for a Dell unit, or at least get the same features at a lower cost if you were to opt for Dell.
Note: I'm mentioning Dell here because that's the only hardware that I support and can compare with. My colleagues tell me that other brands are comparable or better than Dell (notably, Toshiba, HP and Lenovo).
Note to all OSX fans: this is a post by a sysadmin about the warranty and replacement problems with Mac hardware, and not about OSX the Operating System which is well engineered, and which I have some amount of respect for.
Ever since I become a sysadmin at Thoughtworks, I've helped the Studios marketing ensure that the product launch happens successfully.
post a comment
For those who don't know, Marketing created the demand, while Sales turns the leads generated from Marketing into concrete sales.
Each product launch or release has been educational and adventurous. It fun to get a new set of web pages live, and to then watch the httpd logs and find out who's navigating the site at a particular moment.
The Studios Marketing team are like a company by themselves - they decide the targets, then go aim for those targets. There is tremenduous positiveness, a "we can achive" spirit, and the motivation to do a lot of hard work. I intend to complete my present marathon bout of work to go help our Marketing folks with more stuff.
post a comment
That was a highly visited link at cisco.com during the past few months.
I've been really busy creating failovers for just about everything (we have three active SVN mirrors + an S10 based NFS store that gets incremental revision dumps.. Yay!), and now my attention has moved toward the Cisco devices.
Being sysadmin is a fiduciary responsibility - your company, colleagues and customers trust you to do the right thing.
There's so much that we've accomplished, but I've nearly zero time to blog about it all. I intend to complete putting systems in place and setting up failovers for just about everything, and will then hopefully have time blog.