The "one and one" verse is a normal family life. The "two and two" signifies more babies born and the house getting heavy loaded (also a reference to him being booked at clubs)... plus baby cryin. CA and Chicago at the time were the two most successful places in America. Ca. signifying idyllic success and Chicago midwest success as well as a place any musician could get gigs.
The "two and two" verse talking to the "friend -boy" is about the adultery. He is facing his wife's lover. "Two and two" is his normal family, but "Four and two is six" is the mixing up of his family (4) and the wife's lover, and surely his wife also (2). This makes 6. Six is biblically the number of man (and also sin - as only man/humans, not animals, can sin on earth), as all these old bluesmen steeped in biblical stories and dialogue would have known: ((man was made on the sixth day, the sixth sin is "thou shalt not kill", man works 6 days and rests on the 7th, man could only be a slave for 6 years released on the 7th, the number of man, and beast, is 666, man is 6 parts: 5 senses and his spirit, etc etc)). Also now he is cryin', not the baby. And so we have the refrain of taking off to sweet Ca. or Chicago to get away from the problems.
The six and two is eight verse is her cheating with yet another man.. She is now doing this to not Robert, but the man she cheated on Robert with. She's a wicked woman is what he is saying and that cheaters gonna cheat. "Eight and two is ten" is him saying she'll keep doing it. And so we have the refrain of taking off to sweet Ca. or Chicago to get away from the problems. It's a classic blues theme, repeated endlessly. Zep got it... "Going to Calilfornia with an aching in my heart."
The last verse of "I'm going to Ca." signifies he's going alone, not with his wife of her lover that he was advising. And that later they will both separately need his advice again, and that they should of gone with him originally on tour as they are ... "cryin' Hey, hey Baby, don't you want to go Back to the land of California To my sweet home Chicago"