This is usually due to downdrafts, which can be caused by any obstacle that affects airflow around the chimney, such as trees, buildings and adjacent roof lines. Sometimes, adding extra chimney to increase the stack height can help to prevent downdrafts. And some specialty chimney caps can help to prevent downdrafts.
Wood-burning appliances on the lowest level of a home can sometimes be affected by stack or chimney effect. This is when a draft is created in the house due to the difference between inside and outside temperatures that creates uneven air pressure in various levels of the house with a neutral point between them. Wood-burning appliances below this neutral point work against this low or negative pressure while ones above the neutral point are aided by the higher or positive pressure.
Opening a window or having a dedicated outside air connection to your wood-burning appliance might reverse stack effect. You might also consider a residential ventilation system or HRV system to aid in the reversing of stack effect.