
Cross-post from my blog at sun
For one reason or another, many times we find ourselves changing the hostname of a machine. It's been repurposed or moved - or perhaps the original installer didn't know what name it should have. To achieve this on most modern Linux distributions there are 2 key files you need to update.
090825 17:17:15 [ERROR] Failed to open the relay log './mellie-relay-bin.000002' (relay_log_pos 339)
090825 17:17:15 [ERROR] Could not find target log during relay log initialization
mysql> STOP SLAVE;
mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
Master_Host: localhost
Master_User: root
Master_Port: 3306
Relay_Master_Log_File: gbichot-bin.005
Exec_Master_Log_Pos: 79
ERROR 1201 (HY000): Could not initialize master info structure; more error messages can be found in the MySQL error log
mysql> RESET SLAVE;
mysql> RESET SLAVE;
mysql> CHANGE MASTER TO MASTER_HOST="localhost", MASTER_USER="root", MASTER_PASSWORD="test", MASTER_LOG_FILE="gbichot-bin.005", MASTER_LOG_POS=79;
mysql> START SLAVE;
*wait a few moments*
mysql> SHOW SLAVE STATUS\G
mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 8
This week's blog post is not strictly to do with MySQL, but a little more to do with highly available clusters using heartbeat and ldirectord
While many people use heartbeat with MySQL, ldirectord usage is less common in these scenarios and more often using in clustered web and mail servers. It is however sometimes used with a selection of MySQL slave servers.
ldirectord is a tool that manages IPVS in the kernel. Essentially you give it a list of servers that act as a "cluster" for a service, for example, 2 web servers. It will then setup the linux load balancer IPVS to direct to these 2 machines. However, what it does is keeps monitoring the servers, and if one of them goes away, it removes them from the cluster pool.
The problem I have run into a number of times, is that ldirectord gets stuck and stops monitoring services. However when it does this, then services stop getting updated, and if a service goes down, it won't be removed from the cluster. On top of that, I have had it get stuck on START UP, after a failover, in this case not many of the services had a chance to come up yet, and you are stuck with a cluster often with 0 nodes available to service some requests - which causes downtime.
So last night, Shane Short and myself wrote a patch to ldirectord and a nagios plugin, in order to make sure that ldirectord is doing it's job and hasn't got stuck. It works by hooking into ldirectord's '_check_real' function, which is called whenever a server is checked. It will then adjust the timestamp on a 'pulse' file, which we later check with our nagios script.
Here is the patch to ldirectord: http://lathiat.net/files/ldirectord.patch
So now, in the default configuration we will end up with a file at /var/run/ldirector.ldirectord.pulse (this file is actually in the same location as your ldirectord PID file, so if you have moved that, it will be with it) which has it's timestamp updated with each service check.
Now, we make a plugin for nagios, which I have here:
http://lathiat.net/files/check_ldirectord_pulse
And you can configure this plugin into Nagios or Nagios NRPE as normal. You can test it is working by running the plugin manually, you should have a result like this;
OK: ldirectord pulse is regular (4 seconds since last beat)
Greetings Planet MySQL readers,
I am new to Planet MySQL.. so I thought I should introduce myself.
I am Trent Lloyd (some may know me online as 'lathiat'), based in Perth, Western Australia and presently working for Sun Microsystems as a MySQL Support Engineer providing support to Sun's MySQL customers. I have been with Sun for 12 months, and previous to that was working for MySQL AB before it was acquired for 8 months. My background before that is in the ISP industry working for HostAway in a combination System/Network administration and support role.
I have also given a number of papers, often related to either Avahi or IPv6 at a few conferences.. mainly Linux.conf.au - you can view them on my web-site www.lathiat.net.
I have an open-source/free-software community background.. I co-authored Avahi (a mDNS/DNS-SD/Link-Local IP stack for Linux and other *nixes) with Lennart Poeterring and dabbled my hand in the likes of the Ubuntu MOTU and GNOME communities.
Working in the MySQL Support Team, and having a very large customer base to source problems from, we run into many problems and gotcha's on our customer systems which people don't hit every day but are very difficult to track down when you do. I hope that by sharing some of these experiences
With that introduction out of the way, I have just landed in Hobart, Tasmania, Australia to attend linux.conf.au (which was a very last minute arrangement). I have attended this conference every year since 2003 (thats 7 of them!), it's always great and I am glad I am able to attend again.
I am sitting at the "Open Source Databases" mini-conference which has a strong MySQL presence. There are quite a few Sun folk here, which is fantastic - unfortunately a few others were unable to make it but it should be great never the less.
I will be bringing you more Linux.conf.au updates throughout the week and MySQL tips into the future!
I just noticed that as of a few days ago (22nd August) - Avahi is now 3 years old.
Heres my original release announcement:
http://lists.freedesktop.org/archives/avahi/2005-August/000227.html
That's a crazy amount of time, since then I presented at Linux.conf.au 2006 and GUADEC 2007 (Birmingham) as well as some presentations at GNOME mini-confs and the like.
Fun and games... also for anyone following this blog, I regularly twitter as lathiat
http://twitter.com/lathiat
Feel free to follow me, but it tends to be more personal stuff rather than Tech/Open-Source related
Dear World,
Please remind me why I bought a mac?
Love,
Trent
[i was typing away the other day and *blip* my backlight stopped working, permanently, seriously, wtf.]
Howdy all,
I will be traveling to Sydney on Sunday the 20th and 21st - if you're in sydney and want to catch up for a beer or something let me know.
Only timeslot will be Sunday (20th) afternoon as I fly out Monday night (21st)
- lathiat
As most of you reading this are likely aware, I work as a Support Engineer for MySQL AB - which was recently acquired by Sun Microsystems.
If you have been living under some variety of rock-like object, you can see the press release here:
http://www.mysql.com/news-and-events/sun/

Trent Lloyd
MySQL Support Engineer
Sun Microsystems
Howdy all,
I know there are some followers of my blog.. and I know I don't update it very much (sorry!)
However I update my twitter quite regularly, so check out http://twitter.com/lathiat (you can subscribe as an RSS if you want)
Trent
Back home in Perth from Linux.conf.au... couple surprises when I got in.
First up, I made Qantas Silver Frequent flyer. This is mainly a yay for the priority check-in - and I get 1 free Qantas club invitation.
Had I thought about it when I started travelling, and taken Qantas the whole time I probably would have been gold by now but I took 2 big european trips on emirates.. and a whole stack of Virgin Blue domestic. Oh well.
Secondly, I left my air conditioner on.. It's a crappy old wall unit and it has apparently continually frozen and unfrozen up over the week and theres a nice trail of water down the wall onto my mattress which is very damp and has several water stain marks. Obviously been through the cycle a few times. It also managed to eat a bit into the plasterboard under it. -woops-. When I got in it was frozen solid out the filter and nearly out the plastic grating.
Note to self: check this in future.
And last but not least, its too damn hot in Perth.
While I'm in Melbourne for linux.conf.au (which finished yesterday), I thought I'd head down to the Crown Poker Room to get some action.
Unfortunately I missed the Aussie Millions by a couple of weeks, but I did run into one of the final tablist while there. He made alot of money.
I set myself a limit of $300.. so first I tackled the Sunday $100 ($85+$15) NLHE tournament. I didn't fair so well.. I got up in chips early (doubled up).. had a really unfortunate hand..
I was dealt [Kd] exposed, next card was [Ac] and my redealt card was [6s]. I elected to fold and out flopped [Kh][Kc][3d] - I was not happy! I definitely would have been in with [Kd][As] and my trips would have more than likely got me a healthy stack. D'oh.
I endeed up going out just before level 4 with my [Kh][As] vs [Jh][10s]. The flop was a blank and a [10c] came on the river sealing my fate.
From then I moved onto the $2/$3 NLHE. I bought in for $100 and about an hour later was up to $254 and cashed out when my seat in a $50 Sit'n'Go came up.
I didn't do too well in the Sit'n'Go either.. busted out in 6th place calling all-in with my [As][10s] against my offenders [Ah][Ac]. The aces held up.
Then I nearly left.. but decided to head back to the $2/$3 NLHE cash game again for $100. Only about 30 minutes in I was up to $306. At this point although I felt guilty leaving the table a couple hands after winning an $80 or so pot; but I knew it was starting to go to my head and I'd loste up and likely lose it all and probably more back. So I left, $210 up for the day.
Not a bad effort! Tournaments bad.. cash games good.. probably a bit lucky but the tables were quite soft, very often folding to reraises or calling down with bugger all.
Woo!
So having taken my MacBook to the apple store on Tuesday, they replaced the logic board and subsequently found out that not only was the logic board toast, so is the hard drive.
How the hell did I manage that?
So anyway, being 10AM Tuesday morning, and a talk to give at 1:30PM - I cruised down to Myer (by way of a $6 taxi, didn't realise how close it was) and bought an Asus EeePC.
I then rewrote the talk in <2 hours and did it. It was a bit of a rush job but it worked OK.
I'm fairly impressed by the EeePC.. it's got 4G flash, 512MB ram, Webcam.. and comes with a pre-installed and customized Xandros-based OS. It has all the good utils.. Pidgin, Firefox, Skype, Openoffice, etc... although I hope to install Ubuntu on it later.
It did manage to compile a kernel in <hour earlier.. so its not *that* slow.. altho the kernel was a little minimalistic with the features needed for it.
I've really hit a wall with my MacBook.
it has stopped booting after a kernel panic, which was caused by unplugging a Sony Ericsson phone that was just charging.
I can't even boot a cd or into firewire target mode.
This is the 3rd trip now I have had something go wrong with it, dammit. I think life is telling me to go back to a PC and ubuntu!
I guess its
kindof poetic my MacBook running macosx broke at Linux.conf.au.
Posted from my iPod touch.
Well, I made it through my first day in Melbourne for linux.conf.au 2008
Due to horrible travel planning, I arrived in Melbourne at 6am but was unable to check-in to Trinity College until 1PM. Thus I had to wast a couple hours then met some of the organisers and helped shuffle boxes of conference bags out of one room, into cars to then be unpacked down at the venue.
Following that I spent some time folding up t-shirts and finally I was able to register and check in at the college.
Having taken the midnight flight over, I hadn't slept and so I carked it from 4PM to 8PM. This is going to really throw my sleeping out! But oh well..
Following on from my lazing, I located the Trinity common area, with semi-working wireless (some rogue client is handing out DHCP leases that are useless, so you have to refresh 30 times)
So far I have ran into lots of people I know from work and past LCAs.. I'll certainly miss some but they include.. Stuart Young, Stuart Smith, Colin Charles, Giuseppe Maxia, James Iseppi, Arjen Lentz, Joel Stanley, Leon Brooks, Donna Benjamin, Daniel Stone, Grant Diffey, Michael Kedzierski(sp?), Ryan Verner, Tim Ansell.. there are a few people who escape me at the moment!
As wireless is only available in the common area, my 2GB of 3G data with three is certainly proving helpful :)
Until tomorrow... (or well, today)
Heading off shortly to Linux.conf.au 2008 in Melbourne, Victoria.
I've got the red-eye, so I'll be landing in at 6AM Melbourne time - for any other LCAers i'll be staying at the Trinity college.
I'm giving a talk at the MySQL Mini-conf @ 1:30PM on Tuesday - it will be about my experiences as a MySQL Support Engineer and the common sorts of problems that people run into and how to avoid them!
Should be a good week for the conference, and maybe I can lose^H^H^H^Hwin a few dollars at the Crown with my handy dandy poker skills :)
Anyone up for a game? Unfortunately I've misplaced my small, portable and travel-friendly chipset - I really must find that.
Thanks to work, I spent this week in Cupertino, CA. Having never been anywhere in the USA before, it was somewhat exciting to be staying literally up the road from the Apple Campus.
Me @ 1 Infinite Loop
I was also able to goto Google in Mountain View, to attend the Silicon Valley MySQL Users Group at the Visitors Center. Unfortunately I didn't come accross any large google logos to take my photo with, but I did find building number 42
I also spotted the offices of Symantec, Trend Micro, Packeteer, Solid, Microsoft, Borland and MySQL (Surprise!) along the way. Certainly "exciting" for someone thats never been to the bay area.
I had hoped to make it up to San Francisco, and do a little site-seeing of the Golden Gate Bridge and some other stuff, but was too tired to do it this afternoon, and I am attending BarCampBlock today in Palo Alto from early until I fly out tonight - so unfortunately I am going to miss out this trip. Hopefully work will send me back this way again sometime next year.
So, thanks to GUADEC 2007 kindly accepting my talk & paying for my travel expenses, and I had a good time.
I think my Avahi talk went well... we had a good 15+ minute talk about future possibilities, which is what I was hoping.
Lennart's Pulseaudio talk was also very good, it has advanced quite some since his talk @ LCA so it's good to see progress.
I got to meet Sjoerd, and generally the entire Collabora team which was pretty cool - I've fixed some bugs in Avahi so they can progress with the Salut support for OLPC which is good.
I have uploaded my photo collection to flickr, so you can check them out here.
Heres some of my favourites..
I managed to crash the in-flight entertainment system, apparently it uses svgalib..
I love these power transformers that point out all sides, as much as I'm sure they are safe, it just irk's me slightly ;)
Primitive mapping technology...
Walkabout - the Australian Bar - Dude in a kangaroo suit
Me with an iFailPhone
Dress code... no football tops.... exceed england.
www.wtfisthisgame.com
All in all, it was a good trip :)
Just over 18 months ago, I started working at HostAway, initially starting out doing some casual 2-week phone support they needed at the time, my role became permanent and I quickly expanded into both Network & Systems Administration (while still doing a lot of day to day customer support, small organizations tend to demand wider skill ranges :)
None the less I felt, for various reasons, it was time to move on. As such, I have just completed my first week working MySQL AB as a Support Engineer.
The job is home based, and I am currently working the hours of 5AM-1PM local time. Now you may think "wow, you're crazy" for working those hours, but I have discovered that so far I am actually quite enjoying it, I feel I am getting a lot more out of my day at the moment because I seem more awake longer, getting up earlier in the day (Usually about 4:30AM).
I do have the option to start working at 7AM on most days, and given I have no travel time this doesn't put me very far behind a "normal" job as far as messing up the daily schedule.
Being a free software 'person', working for a company with heavy involvement in the free software world is pretty exciting for me. I guess time will tell how it goes, but if this week is anything to go by I think I should be happy in the long term.
In other news, I have discovered that listening to Weird Al Yankovic continually is horrible for your sanity, mostly because he picks alot of very catchy songs to parody, which proceed to persistently circle around inside my head for the next week...
So yesterday I came home to this bird, in my carport
From some googling I *think* it is a pigeon, maybe a dove, but I didnt' check very hard, so I'm far from even moderately sure :) The tag on it's foot says "AUST 2006" and some other stuff I couldn't make out.
It was there all yesterday afternoon, I managed to get quite close to take the above photo, but it did flinch when I tried to get any closer.
This morning, it was still there, and this afternoon when I go thome, it was still there although now its finding its way around the floor.
I do wonder why it likes my carport so much, It doesn't *seem* hurt, and it managed to get up here
So I assume that it can fly somewhat, I guess I'll wait and see if it's there tomorrow... If anyone knows anything feel free to leave a comment :)
I turned on the sprinklers and it was right at the very edge of the big double door out to the world drinking the water splashing in, so it's not stuck in or anything.
Weird!
[for the freedom lovers out there, this it totally and completely unrelated to anything freedom :)]
how the hell did I just lose the following poker hand
I pull pocket sixes, and then flop Ace-6-Ace, i.e. 6's full of Ace's full house, after some serious betting and deceiving I go all in.
HAND
(my hand, circled in black - 6H, 6C, 6S, AH, AS, opponents hand - circled in Red, AH, AD, AH, AS, 6S)
He beat me with 4 of a kind aces.
Opponent Before:
Opponent After:
Note the completely evil look on his face! A conspiracy I tell you...
This after a long string of really stupid pair and occasionally two-pair wins, sheesh!
Watching Jericho S01E14, noticed this...
Looks suspiciously like GNOME 1.4 to me :)
Lennart rightfully pointed out that I didn't really make any conclusion as to the results of my little test, the reason for this is really, "I'm not sure"
Certainly, It seems to be OK, the number of transmitted packets by my rough calculation, make sense, I would be interested to see what the realistic practical throughput of multicast on wireless is when you have many hosts transmitting at once, I know in 802.11b multicast is transmitted at "basic rate" of 1 or 2 mbit (or so I beleive), I'm not sure if 802.11g changes this.
My generally quick gut feeling is "I think this would work" (on wireless), I have no doubt this is fine on a wired network.
More testing to be done...
Talking to sjoerd and others on IRC, (for the benefit of the OLPC project), I decided to attempt to get some kind of an idea of the amount of traffic Avahi generates on a large network.
I booted up 80 UMLs, running 2.6.20.2, on my AMD Athlon64 X2 4200+ (O/C to 2.5GHz per core), with 2GB of ram.
Each was running with 16M ram, a base debian etch install with Avahi 0.6.16.
Interestingly with 80 VMs running my memory usage looked like this:
Mem: 2076124k total, 2012064k used, 64060k free, 18436k buffers
Swap: 996020k total, 8k used, 996012k free, 1476504k cached
I configured a 'UML Switch' with a tap device on the host attached (tun1) and told each VM to come up and use avahi-autoipd to obtain a link-local IP.
I had each VM set to advertise 3 services, via the static service advertisement files
So I was fiddling around with my new phone (Sony Ericcson k800i) and I noticed I could play videos as a ringtone, I was interesting how that worked...
So I downloaded a Music Video of "The Veronicas - When it all falls apart" from the Three Music store, which is at a cost of $3.00 (Which, BTW, came down at 30K/s, not bad for mobile data...)
Once my phone had downloaded it, I had two options "View" (which works fine) and the other was "Ringtone", having selected this the phone stated "This video is restricted against that kind of use"
Sigh.
I wonder what three would say if I asked for a refund ;)
I was watching the movie "Dead or Alive" this afternoon, and I was curious to see the source code scrolling past was from the linux kernel
Interestingly they have blotched out bits and pieces, notably the copyright declaration. They also appear to lack the ability to render tabs.
You can compare to arch/alpha/kernel/err_impl.h (taken from Ubuntu's linux-source-2.6.19), I'll include the excerpt pictured above here:
*
* linux/arch/alpha/kernel/err_impl.h
*
* Copyright (C) 2000 Jeff Wiedemeier (Compaq Computer Corporation)
*
* Contains declarations and macros to support Alpha error handling
* implementations.
*/
union el_timestamp;
struct el_subpacket;
struct ev7_lf_subpackets;
struct el_subpacket_annotation {
struct el_subpacket_annotation *next;
u16 class;
u16 type;
u16 revision;
char *description;
char **annotation;
};
I was toying with the Beta of Redhat Enterprise Linux 5 (RHEL5)
Turns out, it installs Avahi by default, woo hoo :)