Answer:
I've grown potatoes on top of loose earth that's covered by a thick layer of loose straw. It allows the green leafy parts to easily push through to the sunlight above while keeping developing tubers protected from light. I've grown them this way because I really enjoy the taste of fresh new potatoes and have no desire to store mature potatoes for winter. I want to get the most baby potatoes possible while not damaging the plants with tiny potatoes not big enough to eat yet. It also keeps the potatoes cleaner than if grown under the soil.
When I start lifting the straw back to pick the largest new potatoes, I can see the development of the other potatoes. While the biggest potatoes are 1 1/2 to 2 inches in size, others range in size all the way down to 1/4 inch or less.
When a specific garden plant blooms, most of the flowers open within a very short time period. Now think of tomato plants. The flowers continue to open over a period of months and both ripe tomatoes, large and small green tomatoes as well as flowers can all be found on the same plant. Not all the fruit will be ripe before cold weather kills the plant.
Potatoes are closely related to tomatoes and like tomatoes, not all the tubers will be full size when the plant dies back. So I'd say it's neither genetics nor environmental factors but the inherent life and growth cycle of potatoes that account for the tuber size difference.