Phil Robertson of Duck Dynasty made headlines this week because of controversial remarks made about atheists during a speech he made at a Friday prayer breakfast in Florida.

“I find his comments shocking and appalling,” says Eric Adriaans, the national executive director of the Centre for Inquiry Canada, an organization that promotes science, human rights and secularism.

Despite the  nature of his comments, his views on atheists are not unprecedented. According to a survey in 2014 by the Pew Research Centre looking at American’s views on religious groups, atheists were ranked second lowest in positive feelings.

This could be one reason why many atheists are reluctant to express their beliefs, because they fear of being discriminated against, said Adriaans.

“I’ve met hundreds who self-identify as atheists in Canada and the United States … who fear repercussions to their professional careers if it’s known that they don’t follow a faith,” he said.

Adriaans says although Canadians don’t face the same legal punishments as say those in Saudi Arabia for their non-beliefs, the relationship between those in Canada and the Middle East country are not unconnected.

“It’s one of those things that people need to understand that any level of victimization of prejudice is the same thing, not a matter of degree it’s a matter of the same thing.”

Despite the fear of prejudice or discrimination – things are changing for those who identify as atheists.

Eric Thomas is the president of Humanism Canada, and says that the movement towards humanism, which includes atheism, has grown dramatically. According to the 2011 National Household Survey in Canada, roughly 24 per cent of the population considers themselves non-believers, up from 12.6 per cent in 1991.

The experts who were interviewed for this story suggested that education levels correlate with higher rates of religious non-belief. Thomas says, “knowledge is pervasive,” and that people now have the option of looking up information, that once was maybe not easily available.

“The historic paradigm of doing what you’re told because of some book … is going away in a hurry because people look stuff up,” said Thomas.

Education and learning was an influence on Carla-Jean Stokes of Toronto, who identifies as an atheist.  Stokes, 27,  said she started to identify as an atheist over the past few years because she started reading about philosophy and secular humanism.  “Over the past few years I’ve began outwardly expressing those views,” she said. Stokes says she also identifies with being a feminist, and doesn’t think being religious and a feminist are compatible.

Stokes says she fortunately has not experienced much discrimination for her beliefs,  but says it could be because of where she lives.

“When you live in a place like this [Toronto] you can to tell whoever you meet you are an atheist or a feminist, and its okay to have labels like this. In a different setting, you definitely would check yourself before you admit that to other people. It could be dangerous.”

No group whether religious or not should face discrimination, said Adriaans. “Everybody needs the same secular protection to think and believe and speak about a the things they think and believe,” he said.