That video with the flame burning well above the pipe looks like you are making too much vapor for the ammount of air being admitted down at the bottom(running rich). Once the vapor leaves the top of the pipe, it has access to all kinds of fresh air. How can it be smoking when it has access to all the air it can possibly want? Answer: it is being overly cooled:) that and the air cannot reach into/mix with all the vapor before it cools below it's flame point. That is one of the reasons the Babbington burns so completely, it has air(and a lot of turbulence) inside and outside the fireball. That unburnt gas is just going to condense back into carbon in the exhaust and inside the main heat chamber.
Like any fluid that changes states, a tremendous ammount of heat energy is transfered in releasing the vapor(same as boiling water). As an example the ammount of heat lost from an open topped water tank thru evaporation is many many times greater per square foot compared to the heat lost thru the uninsulated wall of the tank for the same surface area. That oil vapor is carrying away a lot of heat before it burns to generate/release more heat. All That flame up there above the tube has nothing to catch or reflect the heat it is releasing from the gas, so that heat dosn't help to warm the incomming air and enhance combustion. Ideally you want all the visible flame to be below the top of the pipe. That way all the heat released helps power the process.
One way you can easilly reduce the fuel vapor produced in a flat bottom pan is to add structure to the bottom of the pan that is tall enough to break the surface of the oil puddle. A handfull of large nuts or ball bearings comes to mind. This will reduce the ammount of vapor produced so you can match it to the airflow.
